Web Excursions 2021-11-07

An Oral History of Bank Python by calpaterson.com

  • Bank Python implementations are effectively proprietary forks of the entire Python ecosystem

    • which are in use at many (but not all) of the biggest investment banks.

  • The first thing to know about Minerva is that it is built on a global database of Python objects.

  • Barbara is a simple key value store with a hierarchical key space.

    • It's brutally simple: made just from pickle and zip.

  • Barbara has multiple "rings", or namespaces,

    • but the default ring is more or less a single, global, object database for the entire bank.

    • From the default ring you can pull out trade data, instrument data (as above), market data and so on.

    • A huge fraction, the majority, of data used day-to-day comes out of Barbara.

    • Internally, Barbara nodes replicate writes within their rings

    • There are some good reasons not to use Barbara.

    • If your dataset is large it may be a good idea to look elsewhere - perhaps a traditional SQL database or kdb+. The soft limit on (compressed) Barbara object sizes is about 16MB. Zipped pickles are pretty small already so this is actually quite a large size. Barbara does feature secondary indices on object attributes but if secondary indices are a very important part of your program, it is also a good idea to look elsewhere.

  • Dagger is a subsystem in Minerva which serves to help keep these data dependencies straight.

    • Dagger tracks the edges in the graph of underlying instruments and automatically reprices derivatives in Barbara when the value of the underlying instruments changes.

    • Dagger is a key technology to get financial models out of Excel, into a programming language and under tests and version control.

  • Walpole is a general purpose runner of jobs, like a mega Jenkins combined with a mega systemd.

    • One real advantage is that Walpole considerably lowers the bar for getting your stuff deployed.

  • Instead of pandas there is a proprietary table library in Minerva: MnTable.

  • It would be wrong to imply that working with any financial software is pure and untrammelled joy.

    • Minerva is no different.

    • New starters take an exceptionally long time to get up to speed - and that's if they don't resign in fit of pique as soon as they see the special, mandatory, in-house IDE (as I nearly did).

    • Even months in, new starters are still learning quite fundamental new things: there is a lot that is different.

  • One of the slightly odd things about Minerva is that a lot of it is "data-first", rather than "code-first".

    • This is odd because the majority of software engineering is the reverse.

    • For example, in object oriented design the aim is to organise the program around "classes", which are coherent groupings of behaviour (ie: code), the data is often simply along for the ride.

    • Writing programs with MnTable is different: you group the data into tables and then the code lives separately.

    • These two lenses for organising computations are at the heart of the object relational impedance mismatch which has caused such grief.

    • The force is out of balance: many more programmers can design decent object-oriented classes than can bring a set of tables into third normal form.

    • This is a large part of the reason that that annoying impedance mismatch keeps coming up.

  • Minerva's "vouch" system, briefly, to another programmer who had never seen it.

    • I explained that when you had a code change, you just had to convince any one of the code owners for the file in question to sign it off.

    • If the change was very urgent, they might sign off your change sight unseen, based on your reputation alone.

    • As soon as they clicked that "vouch" button - bang - your new change was in prod: after all, there is no such thing as a deployment step when your code is stored in a database.

The Grammarphobia Blog: With a Grain of Salt by grammarphobia.com

  • Q: Why “salt” when we “take something with a grain of salt”? Is the salt to counteract something sweet?

  • A: The use of “with a grain of salt” to mean with caution or skepticism first appeared in early 17th-century English

    • as a translation of cum grano salis, a modern Latin expression coined a century earlier.

  • the reason for “salt” here is uncertain, but the earliest English example of the usage that we’ve found suggests that it comes from salting food to make it taste better

  • “The terms of Divinitie are to be taken into the mouth, as the Canonists [canon lawyers] speak, cum grano salis, with a grain of salt, that is, wisely tasted, and understood: otherwise, they will not prove good nourishment.”

On Apps and Coffee – iA by ia.net

  • A cup of coffee ($5 to $10) has a calculable, comparable, and comprehensive value.

  • It’s easy to convince coffee drinkers to pay for coffee.

    • They already want one.

    • To make them happy just hand them over that cup.

    • If you know how to brew a better coffee than Starbucks, which is not a high bar, you can expect some people to come in and order daily.

  • Apps are a different story.

    • The value of an app is a confusing matter.

    • What an app is worth to you can become dramatically personal, contextual, and time-sensitive—

    • it depends on who needs that app—for what, for whom, why, how, when, and by when.

  • Next to free, anything above 0 becomes infinitely expensive and diabolically greedy.

  • Small apps: High quality, fast cycles, real support

  • Big apps: Low quality, slow cycles, no support

  • Unless we’re talking in Java, the comparison coffee:software doesn’t make any sense.

    • Yeah, they both cost money. But so does almost everything.

  • For me, the question is not: How many coffees does an app cost.

    • It’s: How many apps does a coffee cost?

    • So, when I buy a cup of coffee, I do ask myself: ‘How many apps do I need to sell to buy that very cup? 1, 2, 20, 200?’

  • If there is a valid coffee comparison it’s the coffee maker.

    • Productivity apps are not consumed. Productivity apps are tools. They help you make things.

    • Great apps help you make nice things. Pro apps help you make pro things.

    • If you buy a good espresso machine, but you don’t care how to handle it, your coffee will pour out as a thin soup. Same for apps.

  • No one knows and no one cares about the cost of designing, developing, supporting, and managing an app. 

scriptingosx/diskspace: macOS command line tool to return the available disk space on APFS volumes

With the various APFS features the value for free disk space returned from tools such as duor df will not be accurate. This tool uses system functions to get various measures of available disk space.

The 'Important' value matches the free disk space value shown in Finder.

Checking Volume Storage Capacity | Apple Developer Documentation

  • Before you try to store a large amount of data locally, first verify that you have sufficient storage capacity.

  • To get the storage capacity of a volume, you construct a URL (using an instance of URL)

    • that references an object on the volume to be queried,

    • and then query that volume.

  • The query type to use depends on what's being stored.

    • If you’re storing data based on a user request or resources the app requires to function properly

      • (for example, a video the user is about to watch or resources that are needed for the next level in a game),

    • query against volumeAvailableCapacityForImportantUsageKey.

    • However, if you’re downloading data in a more predictive manner

      • (for example, downloading a newly available episode of a TV series that the user has been watching recently),

      • query against volumeAvailableCapacityForOpportunisticUsageKey.