Hackathons we've already built.

Scored on

• Score: Overall score from 0 to 1000: how well fraud is ranked, scaled down for slow answers and failures, on 12 days of payments the service has never seen.
• Fraud ranking (average precision): How well the service puts fraud above normal payments, from 0 to 1. The original service gets about 0.4.
• Fraud caught at 5% review: Share of fraud caught if the riskiest 5% of payments are sent to human review.
• P99 latency: How long the slowest 1% of payments take to score. Target: under 5 ms.
• Median latency: How long a typical payment takes to score.
• Error rate: Payments the service failed to score (it crashed, hung or returned nothing). In production these are approved unchecked.

Scored on

• Score: Overall score from 0 to 1000: how good the top 10 results are, scaled down for slow answers and failures, on two weeks of real customer questions.
• Result quality (nDCG@10): How well the top 10 results are ordered, from 0 to 1 (1 = the most helpful articles first). The original service gets about 0.37.
• First good answer (MRR): How high the first truly useful article ranks. 1.0 means it's always first, which is what the AI answers feature reads.
• Found in top 20: Share of the truly useful articles that appear anywhere in the top 20 results.
• Outdated in top 3: Questions where an outdated or duplicate article ranks in the top 3, and so gets fed to the AI answers feature.
• P95 latency: How long the slowest 5% of searches take. Target: under 50 ms.
• Error rate: Searches that crashed, hung or returned nothing usable: the customer sees an error page.

Scored on

• Score: Overall score from 0 to 1000: 0 is no better than a simple forecast, 1000 is as good as knowing true demand, scaled down for bias and missing forecasts.
• Forecast error (WAPE): Total forecast error as a share of total units sold over the 28 days, counting in-stock days only. Lower is better; the original job gets about 0.43.
• Better than simple forecast: How much smaller the error is than a simple forecast (each weekday's average over the last two weeks).
• Bias: Total forecast compared with total sales. Negative means ordering too little (empty shelves); positive means waste. Target: within ±5%.
• Coverage: Share of products and days that got a valid forecast. The original job skips new products entirely.
• New-product error: Forecast error on products launching during the 28 days, which have no sales history.
• Job runtime (s): How long the whole nightly job took on 2 cores. The limit is 120 s; over it, the job is stopped and scores nothing.

Scored on

• Score: Overall score from 0 to 1000: speed (single messages and bulk), adjusted for the grading machine, times correctness.
• Correctness: Share of checked messages whose answers match the original model exactly (within 0.0001). Below 99.5% the score drops fast.
• P99 latency: How long the slowest 1% of single messages take. The original service: about 250 ms.
• Median latency: How long a typical single message takes.
• Throughput: Messages per second when they arrive in batches of 1 to 64, on one CPU core. The original service: about 9.
• Latency speedup: How many times faster single messages are than the original service, on the same hardware.
• Throughput speedup: How many times more messages per second than the original service, on the same hardware: roughly how much server cost it saves.

Scored on

• Score: Overall score from 0 to 1000: correctness first, then successful checkouts, reliability and speed, on the next sale day's traffic.
• Correctness: 1.0 means payments, stock and what every customer was told all agree perfectly. Money and stock mistakes cost the most.
• Double charges: Customers charged twice (or given two orders) for one checkout. The original service does this when phones retry.
• Oversold units: Units of limited stock sold that didn't exist.
• Pricing errors: Orders charged the wrong amount: rounding, coupon rules, outdated prices or invalid codes accepted.
• Successful checkouts: Checkouts that should have succeeded and did; for limited drops, how much of the real stock was sold.
• Error rate: Requests that failed, were dropped by a crash, or got no answer within the app's 2.5 s timeout.
• Checkout P99: How long the slowest 1% of checkouts take. Target: under 200 ms.

Scored on

• Score: Overall score from 0 to 1000: correctness, adjusted for efficiency, over editing sessions the code has never seen.
• Correctness: Per session, half 'every device agrees' and half 'every edit kept as meant', averaged. Crashed sessions count zero. The original code gets under 40%.
• Devices agree: How often every device ended up showing exactly the same document once all messages arrived.
• Edits kept as meant: Share of checks passed on what users meant: nothing lost or duplicated, deletes stick, typing order kept, later edits win.
• Failed sessions: Sessions where the sync code crashed, hung or ran out of time.
• Work per edit: Processing time per edit, added up across every device.
• Data per edit: Average bytes sent per edit: what the feature costs on a phone's data plan.

Scored on

• Score: Overall score from 0 to 1000: trading cost, staying within limits, reliability and speed, over six trading days the algorithm has never seen.
• Cost vs market average (bps): How much worse than the market's average price (VWAP) the algorithm traded, in basis points (0.01%). Lower is better; the original algo: about 15–20.
• Cost vs arrival price (bps): Difference from the price when each order arrived. Mostly market movement over a few orders; shown for context.
• Completion: Share of ordered shares the algorithm filled itself before each order's deadline.
• Over the volume limit: Share of trading minutes where the algorithm traded more than 20% of the market's volume (the client's limit).
• Orders pulled: Orders the safety switch took away after an error, a bad reply or a hang.
• P99 latency: How long the slowest 1% of market updates take to answer. Target: under 5 ms.

Scored on

• Score: Overall score from 0 to 1000: position and heading accuracy, scaled down for invalid answers and slow updates, on 16 drives the software has never seen.
• Position error (m): Typical distance in metres between where the robot thinks it is and where it really is. The original software is about 30 m off; GPS alone can't beat about 1.2 m.
• Error without GPS (m): Position error while GPS is out (tunnels, overpasses) and for 10 s after it returns.
• Worst 1% position error (m): How far off the robot is at its worst moments.
• Heading error (°): Typical error in which way the robot thinks it's facing. The planner steers on this.
• Invalid answers: Share of position requests answered with an invalid number, an error, or nothing. Each one would stop a robot.
• P99 latency: How long the slowest 1% of sensor updates and position requests take. Target: under 1 ms.