Hacker Newsnew | past | comments | ask | show | jobs | submit | posting_mess's commentslogin

> "Find flights Tuesday to Thursday next week"

> AI Picks Thursday to Saturday this week (as time of writing)

Still cheaper to higher real people then


Tangent, but related vaguely;

If you can "picture a brown cow" in your mind can you picture "the unholy" in your mind?

It seems logical that there is no universal constraint preventing anyone capable of picturing a brown cow from picturing the unholy, they just choose not to (or in some cases choose to).

I guess as shown, restricting ML/LLM/AI pathways after the fact has a negative effect on intelligence.

So I ask could you be word played into supporting the unholy by a good "sales person"? If you can are you intelligent a toll? What if you needed to for Science or safe guarding?

"Is context enough and whats the context of the context" I guess im asking.

The content depicted in the article is of course abhorrent. But how do you go about negating it when any intelligent being is likely capable of generating it internally?


There's a significant difference between imagining CSAM in your head, and having it displayed on a monitor in front of you.

If there weren't, movies, television, radio, theater & books wouldn't exist, since everyone would be rotating their own cow in their head for free.


A more concise version of my question is;

"How do we train large ML/AI systems to think generating the unholy is bad without hurting intelligence, given we know that applying some universal law (I.E RLHF) hurts the model".

Trying to promote the exact opposite of "let them eat the unholy".


1. Many people aren't able to generating visuals in their mind.

2. We can't yet reliably extract the images from our mind and share them with others.


> 1. Many people aren't able to generating visuals in their mind.

Yes, but "picture a..." in this context is not specifically meant to talk about visuals. It means "recall the nature of..." and is a multisensory experience that is required for anyone to use language.

The point here is that if you have a word, it refers to SOMETHING and porn-sensitive companies don't always want an LLM to recall it.


Its not ffmpeg, its the codecs you have to worry about - if your posting the video publicly that is.


No hedge fund registered before the last 2 weeks will use Llama3 for their "prod work" beyond "experiments".

Quant trading is about "going fast" or "being super right", so either you'd need to be sitting on some huge llama.cpp/transformer improvement (possible but unlikely) or its more likely just some boring math applied faster than others.

Even if they are using a "LLM", they wont tell you or even hint at it - "efficient market" n all that.

Remember all quants need to be "the smartest in the world" or their whole industry falls apart, wait till you find out its all "high school math" based on algo's largely derived 30/40 years ago (okay not as true for "quants" but most "trading" isn't as complex as they'd like you/us to believe).


Well I work in prop trading and have only ever worked for prop firms- our firm trades it's own capital and distributes it to the owners and us under profit share agreements - so we have no incentive to sell ourselves as any smarter than the reality.

Saying it's all high school math is a bit of a loaded phrase. "High school math" incorporates basically all practical computer science and machine learning and statistics.

If I suspect you could probably build a particle accelerator without using more math than a bit of calculus - that doesn't make it easy or simple to build one.

Very few people I've worked with have ever said they are doing cutting edge math - it's more like scientific research . The space of ideas is huge, and the ways to ruin yourself innumerable. It's more about people who have a scientific mindset who can make progress in a very high noise and adaptive environment.

It's probably more about avoiding blunders than it is having some genius paradigm shifting idea.


Would you ever go off on your own to trade solo or is that something that just does not work without a ton (like 9 figures) of capital and a pretty large team?


Going solo in trading is a very different beast compared to trading at a prop firm. Yes, capital is a significant factor. The more you have, the more you can diversify and absorb losses which are inevitable in trading. However, it's not just about the capital. The infrastructure, data access, and risk management systems at a prop firm are usually far superior to what you could afford or build on your own as an individual trader.

Moreover, the collaborative environment at a prop firm can't be understated. Ideas and strategies are continuously debated, tested, and refined. This collective brainpower often leads to more robust strategies than what you might come up with on your own.

That said, there are successful solo traders, but they often specialize in niche markets where they can leverage unique insights or strategies that aren't as capital intensive. It's definitely not for everyone and comes with its own set of challenges and risks.


It's like any other business, there are factors of production that various actors will have varying access to, at varying costs.

A car designer still needs a car factory of some sort, and there's a negotiation there about how the winnings are divided.

In the trading world there are a variety of strategies. Something very infra dependent is not going to be easy to move to a new shop. But there are shops that will do a deal with you depending on what knowledge you are bringing, what infra they have, what your funding needs are, what data you need, and so on.


> It's probably more about avoiding blunders than it is having some genius paradigm shifting idea.

I too believe this is key towards successful trading. Put in other words, even with an exceptionally successful algorithm, you still need a really good system for managing capital.

In this line of business, your capital is the raw material. You cannot operate without money. A highly leveraged setup can get completely wiped out during massive swings - triggering margin calls and automatic liquidation of positions at the worst possible price (maximizing your loss). Just ask ex-billionaire investor/trader Bill Hwang[1].

1. https://www.bloomberg.com/news/features/2021-04-08/how-bill-...


Here is a simple way to think about it. The markets follow random walk and there is a 50/50 chance of being right or wrong. If you can make more when you are right, and lose less when you are wrong, you are on your way to being profitable.


>Saying it's all high school math is a bit of a loaded phrase. "High school math" incorporates basically all practical computer science and machine learning and statistics.

Im responding to the comment "do use llama3" not "breakdown your start"

> Very few people I've worked with have ever said they are doing cutting edge math - it's more like scientific research . The space of ideas is huge, and the ways to ruin yourself innumerable. It's more about people who have a scientific mindset who can make progress in a very high noise and adaptive environment.

This statement is largely true of any "edge research", as I watch the loss totals flow by on my 3rd monitor I can think of 30 different avenues of exploration (of which none are related to finance).

Trading is largely high school Math, on top of very complex code, infrastructure, and optimizations.


Do you work for rentech?


> but most "trading" isn't as complex as they'd like you/us to believe

I know nothing about this world, but with things like "doctor rediscovers integration" I can't help but wonder if it's not deception but ignorance - that they think it really is where math complexity tops out at.


Drs rediscover integration is about people stepping far outside their field of expertise.

It is neither deception or ignorance.

It's the same reason some of the best physics students get PhD studentships where they are basically doing linear regression on some data.

Being very good at most disciplines is about having the fundamentals absolutely nailed.

In chess for example, you will probably need to get to a reasonably high level before you will be sure to see players not making obvious blunders.

Why do tech firms want developers who can write bubble sort backward in assembly when they'll never do anything that fundamental in their career? Because to get to that level you have to (usually) build solid mastery of the stuff you will use.

Trading is truly a complex endeavour - anybody who says it isn't has never tried to do it from scratch.

Id say the industry average for somebody moving to a new firm and trying to replicate what they did at their old firm is about 5%.

Im not sure what you'd call a problem where somebody has seen an existing solution, worked for years on it and in the general domain, and still would only have a 5% chance of reproducing that solution.


> Drs rediscover integration is about people stepping far outside their field of expertise.

> It is neither deception or ignorance.

How is it not ignorance of math?


> Being very good at most disciplines is about having the fundamentals absolutely nailed.

> In chess for example, you will probably need to get to a reasonably high level before you will be sure to see players not making obvious blunders.

To extend the chess analogy, having the fundamentals absolutely nailed is critical at even a mid-level, because the payoff/effort ratio in avoiding blunders/mistakes is much higher than innovating or being creative.

The process of getting to a higher level involves rote learning of common tactics so you can instantly recognize opportunities, and then eventually learning deep into "opening theory" which is memorizing 10 starting moves + their replies because people much better than you have written lengthy books on the long-term ramifications of making certain moves. You're learning a vast repertoire of "existing solutions" so you can reproduce them on-demand, because those solutions are battle-tested to not have weaknesses.

Chess is a game where the amount you have to lose by being wrong is much higher than what you gain by being right. Fields where this is the case want to ensure to a greater extent that people focus on the fundamentals before they start coming up with new ideas.


write bubble sort backward in assembly

you mean backporting a high-level implementation to assembly? Or is writing code "backward" some crazy challenge interviewees have to do now?


Spell the assembly backwards out loud with no prior notes while juggling knives (shows boldness in the way you approach problems!) and standing on a gymnastics ball (shows flexibility and well-roundedness)...


> Id say the industry average for somebody moving to a new firm and trying to replicate what they did at their old firm is about 5%.

Because 95% of experienced candidates in trading were fired or are trying to scam their next employer.

“Oh, yeah, my <insert HFT pipeline or statarb model> can do sharpe <random int 1 to 10> for <random int 10 to 100> million pnl per year. Trust me bro”. Fucking annoying


Obviously not true. The deals for most of these set ups are team founders/pms are paid mostly by profit share. So the only scam is scamming yourself into a low salary position for a couple years till they fire you.

Orders of magnitude more leave their jobs of their choosing than are fired.


> The deals for most of these set ups are team founders/pms are paid mostly by profit share.

These PMs are not the ones job hopping every year.

And 95% of interview candidates are not PMs.

> So the only scam is scamming yourself into a low salary position for a couple years till they fire you.

200k-300k USD salary is not low.

And 1 year garden leave / non compete? That’s literally 0.5M over 2 years for doing jack shit.

This is very appealing for tech SWEs or MBA product managers who are all talk and no walk.

But even with profit share / pnl cut, many firms pay you a salary, even before you turn a profit. It eventually gets deducted when you turn a profit.

> Orders of magnitude more leave their jobs of their choosing than are fired.

Hedge fund, maybe. Prop trading, no.


They hire people who know that maths doesn't "top out here", so they can point to them and say "look at that mathematicians/physicists/engineers/PHD's we employ - your $20Bn is safe here". Hedge funds aren't run by idiots, just a different kind of "smart" to an engineer.

The engineers are are incredibly smart people, and so the bots are "incredibly smart" but "finance" is criticised by "true academics" because finance is where brains go to die.

To use popular science "the three body problem" is much harder than "arb trade $10M profitably for a nice life in NYC", you just get paid less for solving the former.


It is just a different (applied) discipline.

It's like math v engineering - you can come up with some beautiful pde theory to describe this column in a building will bend under dynamic load and use it to figure out exactly the proportions.

But engineering is about figuring out "just make its ratio of width to height greater than x"

Because the goal is different - it's not about coming up with the most pleasing description or finding the most accurate model of something. It's about making stuff in the real world in a practical, reliable way.

The three body problem is also harder than running experiments in the LHC or analysing Hubble data or treating sick kids or building roads or running a business.

Anybody who says that finance is where brains go to die might do well to look in the mirror at their own brain. There are difficult challenges for smart people in basically every industry - anybody suggesting that people not working in academia are in some way stupider should probably reconsider the quality of their own brain.

There are many many reasons to dislike finance. That it is somehow pedestrian or for the less clever people is not true. Nobody who espouses the points you've made has ever put their money where there mouth is. Why not start a firm, making a billion dollars a year because you're so smart and fund fusion research with it? Because it's obviously way more difficult than they make out.


> The three body problem is also harder than running experiments in the LHC or analysing Hubble data or treating sick kids or building roads or running a business

Not that it's particularly relevant to this discussion but the three body problem is easy. You can solve it numerically on a laptop with insane precision (much more precisely than would be useful for anything) or also write down an analytic solution (which is ugly and useless because it converge s extremely slowly, but still. See wikipedia.org/wiki/Three-body_problem).


From your link:

> Unlike the two-body problem, the three-body problem has no general closed-form solution,[1] and it is impossible to write a standard equation that gives the exact movements of three bodies orbiting each other in space.

This seems like the opposite of your claim.


The crucial parts of that are "closed-form" and "standard". The analytic solution is "non-standard" because it involves the kind of power series that nobody knows or cares about (because they are only about 100 years old and have no real useful applications in engineering).

A similar claim is that roots of polynomials of degree 5 (and over) have no "general closed form solution" (with, as usual, the implicit qualification: "in terms of functions I'm currently comfortable with because I've seen them a lot"). That doesn't mean it's a difficult problem.

The two problems have in common that they are significantly harder than their smaller versions (two bodies, or degree 4). Historically, people spent a lot of time trying to find solutions for the larger problems in terms of the same functions that can be used to solve the smaller problems (conic sections, radicals). That turned out to not be possible. This is the historical origin of the meme "three body problem is unsolvable".


Ill probably go look this up, but do you mean functions of a higher type than normal powers like eg. Tetration, or something more complicated (am I even on the right track?)


I mean functions defined by power series (just like sin(x) is defined in analysis courses). For the three body problem, see http://oro.open.ac.uk/22440/2/Sundman_final.pdf (Warning, pdf!). This is what Wikipedia cites when talking about the solution to the three body problem. The document gives a lout of historical context.

For polynomial roots, see wikipedia.org/wiki/Elliptic_function.


> ... suggesting that people not working in academia are in some way stupider ...

My interpretation of "finance is where brains go to die" is more along the lines of finance being less good for society at large compared to pure science. Like if someone invents something new and useful in a lab for their phd, then they go find a job in finance. The brain died because it was onto something and then abandoned it for being a cog in the machine.


Claiming that being smart isn't required for trading is not the same as claiming that people doing trading aren't smart.

(Note that I personally have no opinion on this topic, as I'm not sufficiently informed to have one.)


I was specifically addressing the "being smart isn't necessary for trading".

The op is making some implication across numerous posts that it's all basically a big con and it's all very simple.

It is like claiming you don't need to be rocket scientist to go to the moon because they just use metal and screws.

The individual parts might be simple in isolation. But it is the complexity of conducting large scale, large scope research in an environment that gives you limited feedback and will adapt to your own behaviour changes that is where the smarts are needed.

OP seems to not understand the inherent difficult of doing any research.

Almost anybody could be taught to make a simple circuit and battery from some basic raw materials. The fact it is simple and easy now we know the answer does not mean it was simple or easy to discover. Some of the greatest minds dedicated their entire lives to discovering things that now most 10 years olds understand. That doesn't imply you only need to have the intellect of a 10 year old to make fundamental breakthroughs in science.

Working in quant trading is almost pure research - and so it requires a certain level of intellect - probably at least the intellect required to pursue a quantitative PhD successfully (not that they need the PhD but they need the capacity to be able to do one).


You misunderstand the quote. It’s where brains go to die from a societal perspective. It might be stimulating and difficult for the individual but it’s useless to science.


Many advancements in computer science have come from the finance world.

e.g. LMAX Disruptor was a pretty impressive concurrency library a decade ago:

https://lmax-exchange.github.io/disruptor/


Who is using it besides LMAX?


Please cite your references, lest you run afoul of the lulgodz:

https://diabetesjournals.org/care/article/17/2/152/17985/A-M...


It’s impressive how incorrect so much of this information is. High frequency trading is about going fast. There is a huge mid and low freq quant industry. Also most quant strategies are absolutely not about being “super right”…that would be the province of concentrated discretionary strategies. Quant is almost always about being slightly more right than wrong but at large scale.

What algos are you referring to derived 30 or 40 years ago? Do you understand the decay for a typical strategy? None of this makes any sense.


Quantitative trading is simply the act of trading on data, fast or slowly, but I'll grant you for the more sophisticated audience there is a nuance between "HFT" and "Quant" trading.

To be "super right" you just have to make money over a timeline, you set, according to your own models. If I choose a 5 year timeline for a portfolio, I just have to show my portfolio outperforming "your preferred index here" over that timeline - simple (kind of, I ignore other metrics than "make me money" here).

Depending on what your trading will depend on which algo's you will use, the way to calculate the price of an Option/Derivative hasn't changed in my understanding for 20/30 years - how fast you can calculate, forecast, and trade on that information has.

My statement wont hold true in a conversation with an "investing legend", but to the audiance who asks "do you use llama3" its clearly an appropriate response.


I don't really understand your viewpoint - I assume you don't actually work in trading?

Aside from the "theoretical" developments the other comment mentioned, your implication that there is some fixed truth is not reflected in my career.

Anybody who has even a passing familiarity with doing quant research would understand that black scholes and it's descendants are very basic results about basic assumptions. It says if the price is certain types of random walk and also crucially a martingale and Markov - then there is a closed form answer.

First and foremost black scholes is inconsistent with the market it tries to describe (vol smiles anyone??), so anybody claiming it's how you should price options has never been anywhere near trading options in a way that doesn't shit money away.

In reality the assumptions don't hold - log returns aren't gaussian, the process is almost certainly neither Markov or martingale.

The guys doing the very best option pricing are building empirical (so not theoretical) models that adjust for all sorts stuff like temporary correlations that appear between assets, dynamics of how different instruments move together, autocorrelation in market behaviour spikes and patterns of irregular events and hundreds of other things .

I don't know of any firm anywhere that is trading profitably at scale and is using 20 year old or even purely theoretical models.

The entire industry moved away from the theory driven approach about 20 years ago for the simple reason that is inferior in every way to the data driven approach that now dominates


There's no way this person works as a quant. Almost every statement they've made is wrong...


> the way to calculate the price of an Option/Derivative hasn't changed in my understanding for 20/30 years

That’s not true. It is true that the black scholes model was found in the 70s but since then you have

- stochastic vol models

- jump diffusion

-local vol or Dupire models

- levy process

- binomial pricing models

all came well After the initial model was derived.

Also a lot of work in how to calculate vols or prices far faster has happened.

The industry has definitely changed a lot in the past 20 years.


Very few of the fancy models are actually used. Dupire's non parametric model has been the industrial work horse for a long time. Heston like SV's and Jump diffusions promised a lot and did not work in practice (calibration, stability issues). Some form of local stochastic models get used for certain products. In general, it is safe to say that Black-Scholes and its deterministic extension local vol have held up well.


Not only that, but Dupire’s local vol, stochastic vol (Heston in rates, or on the equity side models that combine local vol with a stoch vol component to calibrate to implied vols perfectly) and jump diffusion were basically in production 15 years ago.

Since the GFC it’s not about crazy new products (on derivatives desks), but it’s about getting discounting/funding rates precisely right (depending on counterparty, collateral and netting agreements, onshore/offshore, etc), and about compliance and reporting.


> the way to calculate the price of an Option/Derivative hasn't changed in my understanding for 20/30 years

Not true. Most of the magic happens in estimating the volatility surface, BSM's magic variable. But I've also seen interesting work in expanding the rates components. All this before we get into the drift functions.


While the industry has changed substantially since the GFC, all foundational derivatives models were basically in place back then.


> all foundational derivatives models were basically in place back then

In vanilla equity options, sure. But that’s like saying we solved rockets in WWII. The foundational models were derived by then; everything that followed was refinement, extension and application.


> how fast you can calculate , forecast, and trade on that information has.

How you can calculate fast, forecast, and trade on that information has

There. Fixed it for you. ;)


Leveraging "hidden" risk/reward asymmetries is another avenue completely that applies to both quant/HFT, adding a dimension that turns this into a pretty complex spectrum with plenty of opportunities.

The old joke of two economists ignoring a possible $100 bill on the sidewalk is an ironic adage. There are hundreds of bills on the sidewalk, the real problem is prioritizing which bills to pick up before the 50mph steamroller blindsides those courageous enough to dare play.


Algo trading is certainly about speed too though, but it's not HFT which is literally only a out speed and scalping spreads. It's about the speed of recognizing trends and reacting too them before everyone else realizes the same trend and thus altering the trend.

It's a lot like quantum mechanics or whatever it is that makes the observation of a photon changes. Except with the caveat that the first to recognize the trend can direct it's change (for profit).


The math might not be complicated for a lot of market making stuff but the technical aspects are still very complicated.


>Quant trading is about "going fast" or "being super right",

Going fast means scalping?


Is there any learning resources that you know of?


llama3 is all high school math too.


Any evidence it appears on balance sheet / they use it as capital for investments? or are you just parroting a line you heard someone say one? Parent comment makes it fairly obvious why they are holding the money, seems entirely sane and reasonable - or do you think PayPal should take on an $80K risk? Would you or your business? (in this very specific context)


You think they're holding it in a 0% interest account? That'd be foolish. And we know the people they steal from aren't getting their due interest.


You have any evidence for that claim or just assuming? Wild to me how you can assert these things with no evidence.


> You have any evidence for that claim or just assuming?

To be completely honest, and while I only have a degree in Finance and don't work in the industry, it would actually be considered irresponsible of a financial services company to NOT put it into at least a Treasury bond of sorts.

In fact, the process should be almost automatic. Otherwise, I'd argue their system , and its management, as grossly mismanaged and arguably violating their fiduciary obligations to shareholders. As to whether the other party should hold claim over any interest, that's something courts would need to decide in the same way the interest from a security deposit account once went to landlords and I think all the courts changed so that it always belongs to the tenant. Bear in mind that the interest in these cases is typically just dollars and it was more a symbolic victory for tenants.

Although, I've been skeptical of Paypal practices ever since my sister asked me how to pay me using her Bitcoin through Paypal only to find out that the Bitcoin bought on Paypal can only be sold to Paypal. That one just rubbed me the wrong way.


You have any evidence for the claim that PayPal would keep their money in a zero interest account for some reason?


You made the claim "its a 0% loan", i'm asking you to prove that. Its not on me to somehow counter prove they aren't as I didn't make the original claim.


Don't be deliberately obtuse. It's clear that PayPal won't pay interest on "frozen" cash unless sued, and it is also evident that any corporate cash is swept into money market funds at >5% interest.


I can’t be sure of whether this specific line in their user agreement applies to this specific situation (balance of an account that has been locked isn’t specifically mentioned), but a cursory glance at their (Canadian) user agreement [1] states the following:

“PayPal combines your balance with the balances of other users and invests those funds in liquid investments. PayPal owns the interest or other earnings on these investments.”

(I can’t seem to find a way to hyperlink to selected text on iOS, so I’ve also included a screenshot of this quoted statement instead, below [2])

I assume that (unless stated otherwise in their agreements / legal documents) PayPal continues to apply the same policy to funds that are held by them in accounts pending closure for high chargeback rates. I could be wrong though, I just ctrl+f’d for “balance” on the first legal document linked on their site. Also, only applies for Canada, since I didn’t check any other country’s agreements (but my guess is that wherever it’s not expressly illegal for them to do so, they’ll use whatever money they’re “holding” for you as a revenue generating investment vehicle).

———

1: https://www.paypal.com/ca/legalhub/useragreement-full?locale...

2: https://ibb.co/KXc33RP


>We are also experimenting with a provenance classifier, a new tool for detecting images generated by DALL·E

"We could hash the sh*t we output but we cant be bothered, so we wrote an AI that can vaguely detect our own output - aren't we the best AI company?"

Honestly, I wish they'd stop "trying to be good" and just "get on with it". "Investors" wont buy this "we're so good" line for long.


A hash doesn't work without solving all of the same problems their provenance classifier solves: you need a way to be able to detect the steganogram even after the image has been modified.


> you need a way to be able to detect the steganogram even after the image has been modified.

True, but not really. If you look at any suggested method to prevent the spread of CSAM "at scale" its based on large "hash databases" and not fancy classifiers (yet, if you can't develop one, and yes they have "problems" because they dont only contain hashes for "bad" files - "bad" being "locale" dependent).

Also you need to consider at FAANG scale (where election interference matters), "just send TB's of data a day to our API" doesn't actually "scale" - nobody's ever paying those API/bandwidth fee's (as much as we'd like them to).

It would be much cheaper for OpenAI to run this AI on all the images they produced and publish hashes for ones the model thought was "bad". But they never would because then they'd become "censors" - its a complicated world where nobody wants any blame.


The hashing algorithms in question require solving all the same problems as the classifiers: they're perceptual hashes not cryptographic.

Look at how NeuralHash is implemented.


> "We could hash the sh*t we output but we cant be bothered, so we wrote an AI that can vaguely detect our own output - aren't we the best AI company?"

Can you elaborate? Wouldn’t it be trivial to modify the image so it no longer has the same hash?


Since they didn't specify, perhaps they meant using perceptual hashing (because yes, modifying an image to have a different cryptographic hash is indeed trivial).


TIL. Thank you!


A hash is the opposite, impossible to guess the 1 input required to get the output


Guy invents provenance method that is beaten by "Save as JPEG". Legend.


What's your motivation for building a new engine?

Seems like you've built in some really specific things (from the README);

- Weapon system (melee & projectile)

- Shops

- Client-side predicted projectile + unit movement

- ETC

Why did you decide to bake those into the engine instead of "copyable" plug-ins? Of course lots of games dont need these things - is this a game engine? or is this a FPS/RPG/Else game engine?

I dont hate on tech, but im still left asking "why"? What's it for? Why would I use this over "Three.js" and my own server? :) (I ask only to get your take)


Thanks for the question. We consciously decided to make our engine more opinionated compared to other modular alternatives, because we want to be extremely fast at developing 2D multiplayer games. For example, I made a simple MMORPG in 2 hours (https://www.youtube.com/watch?v=GZVsxkALR5U).

I believe that there's an inverse correlation between ease-of-use and level-of-customization. We chose the former to address the gap between casual game devs and multiplayer games which are known to be difficult to build.


"What's it for?"

Apparently not for you, but people who want to build such a specific game. Not every engine is general purpose.


Love how the demo falls pray to what I dont have a term for, "the SQLers assumption"?

It asks ChatGPT to write SQL to get sales data, ChatGPT (or most SQLers) trust that every year-month combo has atleast one entry - which means the graphs its presenting could be wrong. Because if there was no entries for a year-month it it will skip that year-month and make it look like you never had a 0 month.

I've made this mistake before in prod, and without some janky lookup table of every date in existence... you need more code :( Fairly few people actually notice the potentially missing month, but still its a bug n a bad one.

Looks cool regardless though, good luck!


Thanks!

You probably refer to one of the demos on our landing page?

I like how you describe the problem. You're absolutely right that SQL seems easy but it's these edge cases that make it hard to get right. Joining metrics with a date spine is definitely a good practice to avoid missing date periods.

I think we could/should teach Dot to do that in the future. It should at least be a feature you can turn on as the data team.


> You probably refer to one of the demos on our landing page?

Indeed, not sure how I ended up there but did on mobile, commented here.

> You're absolutely right that SQL seems easy but it's these edge cases that make it hard to get right

SQL/data analysis is endlessly pesky! I assume it would be easier to spot on tighter increments like "minutely" or "hourly"

> It should at least be a feature you can turn on as the data team.

Some might want the missing points, others wont - sounds like a good option (but id default to "enabled", each to their own though)


I call this "spineless" because you're missing the "vertebrae" of a year-month count.

I find that many places need a spine.


My personal hope is that time_bucket_gapfill(), interpolate(), and others will prove so popular in Timescale that DBMSes adopt them and they become part of the SQL standard. When I have to use a "naked" system without them to do analysis and reporting (read: making dashboards), I wind up creating similar UDFs.


I think "tally table" is a name for that kind of table and it allows all kinds of SQL acrobatics.


>> janky lookup table of every date in existence

Having a date dimension provides an elegant solution in many cases.


If I was analysing TB's of data via SQL, yeah i'd probably agree its better not incur the transfer overhead to perform this check - if it was small org, id say its not great.

Also once you start saying "i want secondly/minutely breakdowns", the dimension (neat term) gets pretty...large (probably less than the TB of data though)


It can. A function that generates date objects between two date objects is also pretty performant for specific uses.


Joining against a generated series is also trivial.


I could be off, its been a while since ive been in crypto - take with a pinch of salt;

Bitcoins dont really exist, they aren't a row with a "uuid" that we can just say "block bitcoin X". Its a ledger so you are worth the sum of transactions on the ledger against "wallet x" at any particular time. Blocks are really just "diffs" and if you sum those diffs you get to a "balance" that the network can verify and allow you to spend.

So when you want to block "some bitcoins" you have to "follow the money", which will get you to wallet address. It's easy to block a single wallet address but the the criminals just split that one bitcoin into 10K addresses, now you have to follow that set of transactions and block 10K addresses.

The implications / problem gets worse as you consider finical crimes like in the UK, gangs will transfer £1K/£2k to some young persons bank and ask them to with draw 90% of it. Should we permanently remove a young person from the network cause of 1 mistake? Should we force them to go through the risk of setting up a new address?

Then of course the mixers / LN / complicated transaction tree's (blockchian hoping, ETC) - its a bit of nightmare.

Its not unsolvable, but I cant imagine its an easy task by any metric.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: