On Prices

The upcoming API 0.0.8 release targets Global Units, Custom Units, and a new Price object that bridges them. It is forcing me to consider some difficult questions.

Amatino’s double-entry data model adopts a ‘quantum mechanics’ approach:  Transactions are the atomic units of all financial information, but they themselves are composed of the more fundamental Entry object, with which we never directly interact. This approach allows financial data to be stored as arbitrary precision numbers in their native denomination.

It also means that the data model does not inherently satisfy the double-entry accounting equation. If external constraints are not applied, it would be possible to store an unbalanced Transaction: The composing Entries may not add up to zero, and the equation would not hold.

Obviously, Amatino is designed with stringent constraints. Attempt to insert an unbalanced Transaction and the digits ‘400’ will present themselves quick smart. But what if a Transaction balance is changed after it is inserted?

How could this happen? It is a matter of relativity. Say you create a Transaction (TX) worth 1 U.S. Dollar (USD) with Entries party to two Accounts: One denominated in USD, and one denominated in Woozlewozzles (WZL). WZL being some Custom Unit you have fed to Amatino.

Perhaps TX is created on 1 Jan 2018, dated at 1 Jan 2018, and at that time, the last WZL / USD price you provided was for 31 Dec 2017, at 2 WZL = 1 USD. Amatino will eat TX and store one Entry for 1 USD, and one Entry for 2 WZL.

Now suppose that on 2 Jan 2018, you have access to new price data. You supply Amatino with a 1 Jan 2018 WZL / USD price of 1 WZL = 1 USD. Ceteris paribus, TX is now unbalanced: Attempt to retrieve TX and Amatino will compute output using the price with time closest-to-but-less-than TX time: 1 Jan 2018.

That yields a WZL entry with double the value that it had at Transaction entry. The double-entry equation is violated and the universe (presumably) implodes.

What’s the solution? The two I am working with right now are:

1. Anchors

When creating a Transaction, a client may specify some (but not all) constituent Entries as being ‘anchors’. When a new Price is inserted, Amatino looks for any Transactions with Entries party to an Account denominated in the Priced unit, with an effective time less than the next most recent Price, and greater-than-or-equal to the new Price.

Amatino then calculates the difference between the old balance and the new balance based on the new Price. That difference is distributed proportionally across any (or the only) Entry that is not marked as being an anchor.

One advantage of Anchors is that they would be compatible with future-dated Transactions with heterogeneous Global Unit denominations.

2. Blind Repair

Similarly to solution (1), Blind Repair would look for any Transactions affected by a new Price. Instead of applying the difference in balance to specially identified Entries, Amatino applies the difference to any Entry in the Priced unit.

One advantage of Blind Repair is that it does not add new complexity to the Transaction model. Anchors would require implicit or explicit decision making on every Transaction entry.

Blind Repair would not be compatible with future-dated Transactions with heterogenous Global Unit denominations. Global Unit prices are updated metronomically at ~1800 UTC each trading day. If a Transaction contained Entries party to say, USD and Euros, how would Blind Repair choose which Entries to adjust? I don’t see a way to make the process deterministic.

What to do?

Right now, I am leaning towards choosing Blind Repair. I would combine it with a new restriction: Transactions cannot be future-dated when they are denominated in heterogenous units where one of those units is a Global Unit.

It will be frustrating to be unable to conveniently future-date known upcoming or recurring Transactions with such denominations. However, I think the increased complexity of Anchors means they are not a viable alternative.

Have thoughts on a third way? Let me know in the comments!

– Hugh


Alpha FOREX limitations on future dated transactions

Amatino was designed to be utterly cavalier about currencies. The end goal was something like this:

Store a transaction in U.S. Dollars, retrieve it in Pounds Sterling, and balance a ledger with it in Euros.

… All without any effort on the part of the user. No special syntax, no premium plans, no loss of fidelity, no fuss. For the most part the Amatino Alpha achieves this goal.

However, there is one important circumstance under which the API will currently throw a ‘501 – Not Implemented’ error. That is when you attempt to create a transaction using mismatched currencies, when prices for those currencies are not available.

For example, let’s say you create a transaction with an effective date of 11 July 2018 (in the past). Perhaps that transaction hits a USD account with a debit, an AUD account with a credit, and you denominate the whole thing in GBP. Amatino won’t miss a beat. Your transaction will be stored. Hooray!

Now, the 501 case. Say you create a transaction with effective date of tomorrow. Amatino doesn’t yet have prices available for USD, GBP, and AUD. It will 501 and you will be a sad camper. If you wait a day for Amatino to receive prices, and then again attempt to execute the transaction, it will go through.

I don’t expect this situation to last for very long. The reason for the limitation isn’t some insurmountable technical obstacle, it’s just time. Amatino has been in development for a long time, and I think it would be unhealthy to keep it private any longer.

Being able to process future dated transactions in mismatched currencies was simply a feature I’ve decided to implement after Alpha release.

Note that this limitation does not affect Custom Units. With Custom Units, all bets are off. Amatino won’t stop you adding future-denominated transactions in Custom Units, but it won’t update them on price changes, either.

I’ll be removing this limitation as quickly as possible. To be notified when the job is done, sign up to the Amatino Development Newsletter.

– Hugh

Availability & Performance Monitoring

If an API explodes in a forest, does it make a sound?

The scariest thing about launching Amatino is the idea that people might try to use it, and be met by a stream of errors. That is not true. It is not the errors that scare me: Errors are an inevitable part of development.

No it’s not the errors, it is the idea that I might not know the errors are occurring. Under the hood, Amatino does all the error handling, collating, and reporting that one would expect. But I wanted a way to see high level API health at a glance.

Enter the Availability and Performance pages. These pages give a quick snapshot of both what proportion of requests are succeeding (Availability), and how fast they are being served (Performance).

Amatino API performance histogram
Performance histogram available at https://amatino.io/monitoring/performance

If these things are going to be monitored, they might as well be public. I would like to publicise as much data about the Amatino API as possible, in order to build trust in the system.

Amatino API availability summary
Availability summary available at https://amatino.io/monitoring/availability

On my future wishlist are graphs describing total system load. For example, memory and CPU usage, cache hit/miss ratios. I’d also like to publish graphs breaking down request time by type.

If you have an idea for data you would like to see, please let me know in the forums!

During alpha, the billing system will run like ass

Update 27 Jun, 18 – The Amatino Alpha servers have been relocated to N. Virginia, USA. The billing system no longer runs like ass! But it will still be buggy. If you find a bug, email me! hugh@amatino.io

Original article posted May 28, 18:

Amatino’s billing system is super flexible. You can switch seamlessly between plans, billing currencies, and numbers of users. You can choose to be billed by API request or by seat. Everything is pro-rated.

And it runs terribly.

Almost every page in the billing system takes more than a second to load. It’s awful.  This awfulness will only persist during the alpha stage of Amatino development. Once we hit beta, the billing system will perform much more responsively.

We can blame the slow alpha billing system on the speed of light. Amatino’s alpha servers are located in Australia. Underlying the billing system is Stripe, a slick payment processor based in the US. Every time you try to visit a billing page, Amatino’s Australian servers strike up a conversation with Stripe. With more than 200ms latency, and three round trips per HTTPS request, things slow to an awful crawl.

I considered stripping all the Stripe requests into asynchronously loaded interface elements, but decided that a little bit of alpha stage pain was acceptable, and that efforts were best expended elsewhere.

So, I’m sorry for the slowness. Once Amatino’s beta servers come on-line, the latency to Stripe will drop to the single digits and the billing system will speed up.

Constructing Amatino’s Alpha Infrastructure

The first working version of Amatino was a pretty simple piece of software. It ran on a single server, and could be utilised via HTTP requests or a bare-bones MacOS Cocoa application.

Amatino “V1” was born in November 2016. It was the product of about two years of research into accounting data-structures. As soon as I finished V1, I realised it was nowhere near being a useful service for a wider audience.

Useful services are highly available, respond with low latency, process requests quickly, and are shielded by strong security. Sure, I could throw more cores and RAM at V1, but the reality is that to provide those aforementioned attributes, a service needs to scale out across an arbitrary number of machines.

Amatino “V2” takes the V1 core R&D and turns the volume up to 11. Amatino’s V2 is to V1 as SpaceX’s Falcon 9 is to Falcon 1. Falcon 1 proved concepts, but Falcon 9 provides a useful service.

In V2, concerns that could be combined on single machines are separated: For example, database machines don’t run caches, and servers providing the website don’t process core API requests. Whereas V1 could run on a single server, V2, at a minimum, requires a constellation of six. And, to be properly tested, it should be run across thirteen.

Spinning up six or more machines on a cloud provider would cost a pretty penny. I don’t know if you are going to like Amatino, or how long it will take before someone might generously choose to subscribe. It would not be prudent to spend big-bucks on hosting at this stage.

Instead, I repurposed hardware from an old gaming rig. The V2 Alpha runs across thirteen virtual machines on an Intel 5820K CPU, with 32GB of RAM, backed by a Samsung 960 Pro NVMe disk, connected to you by a consumer-grade fibre line.

Hardly the hardware a production-grade service should be running on, but perfect for a minimum-viable product.

I call the machine ‘Starport’. The worst part about running the Alpha on Starport is that is located in Sydney, Australia, which is the ass-end of the universe when it comes to network latency. Everyone that connects to Amatino is going to get a ping of one gajillion.

In some ways, that’s good: It will force the software to operate under worst-case latency conditions. If Amatino can provide good service at 400ms, then it should provide a great service at 20ms.

20ms really is the final goal. V2 has been designed to spin up global branches in seconds. Each branch then serves all requests in closest geographic proximity, while retaining global data consistency. Over the coming months, I want to spin up instances in proximity to interested users (hey, maybe that’s you!), and gradually move all processing off Starport.

If you would be interested in testing an Amatino branch near you, let me know your whereabouts in the discussion forums. I’d love to spin one up for you so you can give me feedback!

– Hugh