Pandas vs. Julia – cheat sheet and comparison

This is very similar to DataFramesMeta:

https://github.com/JuliaData/DataFramesMeta.jl

I think having intermediate variables is sort of 'littering', and requires extra work in the naming which might not be necessary. Also, with pipes, you can just take out any intermediate step by commenting out a line or deleting it. You cannot do this with your method above without then going and rewriting many different arguments. I also like piping because you can quickly increment and build a solution - quicker than naming intermediate steps anyway.

Naming intermediate steps require some non-trivial efforts. It can even distract from the main task of getting the results.

In programming the code will be read multiple times and good names will help the future readers. But in data science the calculation will be most likely will not be reused. So efforts to name things will be waste of time.

I suggest trying to lean into it more

I suggest that trying to strictly only bind output to a symbol if it will be used in multiple places.

So when I read code and I see some "intermediary" value bound - it tells me immediately "this thing will be used in several spots". Thereby bindings actually start to convey extra information

Anyway, it's just something that's worked for me. In all other scenarios I will use threading/pipeline (maybe Clojure specific). If steps are confusing/complex then you make a local named lambda or add in the extreme case.. comments

If nothing else, you can just pipe the code and then write comments explaining what's left after each step. But the verbose code can be substantially slower (which happens when piping can be used to perform all these operations lazily).

Did you try v1.9 or v1.10 yet? From others I'm hearing that the code caching changed Makie from about 70 seconds down to 10 in v1.9, and then the loading time improvements brought it to like 5 (unreleased of course, though v1.10 should be branching in a few weeks). Makie load times were of course one of the ones highlighted in the release notes of v1.9: https://julialang.org/blog/2023/04/julia-1.9-highlights/. So while Makie won't be "instant" by v1.10 (<1 second), it was one of the worst offenders before and has gone from "wtf" to "bad but manageable".

A neat solution, but you can’t alter the position of the argument per function.

Yes, but try using this and then try Julia's way. I tried this pandas implementation once and never touched it again.

The flip side is that in pandas, chaining is less uniform because it is based on methods.

In R you can pipe a data frame into any function from any package or one you just wrote, so you use %>% for any piping that happens. In pandas, you have special pandas methods that don't need the pipe, but to pipe with any other function, you have to write .pipe.

The comparison is not really between %>% and ., it's between "you just use %>% for everything" and "you use . for a bloated, somewhat arbitrary collection of special pandas methods, and .pipe for everything else".

What do you mean by wrapping the command in ()? I haven't seen this before. Do you have a link to where they mention this in the docs?

I agree with your conclusion but want to add that switching from Julia may not make sense either.

According to these benchmarks: https://h2oai.github.io/db-benchmark/, DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each.

For what it's worth, data.table is my favourite to use and I believe it has the nicest ergonomics of the three I spoke about.

Just make sure you find the appropriate documentation because the package changes it's syntax an awful lot over the past four years or so and there are lots of tutorials, videos, and blogs that don't apply anymore.

Similarly make sure you research the ecosystem because everything in Julia is very fragmented, IE pandas.loadcsv will require two or more packages in it's Julia equivalent.

Any examples? I've found Julia far easier for simple things than python. Most modern problems are mathematical and nature and I think it's pretty objective that julia looks closer to the mathematics. ----Some tests of very simple tasks in both languages.

At its most basic the obviouses becomes different:

Let's try to get a very simple object: a 3,3 matrix of random booleans in both languages: Julia:

A = rand(Bool,3,3)

Python: No standard support for Matrices. I could really do it a disservsice and comapre the "core language", but that''s obivously stupid so we'll bring in some external libraries to make it easier. Of many ways to skin the cat here's one.. Python

import numpy.numpy as np gen = np.random.default.default_rng() B = gen.choice([True,False],(3,3))

BTW julia has this choice function built into the command as well, so rand(["Which", "Word", "Will", "I", "get?"]) produces exactly what you'd expect.

----

Acutally I can't think of any cases at all off the top of my head. Sorry, I mean my np.somenamespace.another.namespace.sparce head :) I mean just going down the list of things that make code easier in Julia..

* Python requires a third party library for any kind of linear algebra. and matrix multiplication:*

  A =[1 4;6 7"; B = [2 ;3]; A*B doesn't work, i.e. you literally even multiply a matrix! This is madness.you'll need yet again a third partly library

  import numpy as np
 import math #sigh
      x = np.array([1, 2, 3, 4, 5])
      f = lambda x: sin(x)

Even basic string interpolation in Julia is a much nicer "trivial task" than in python alone

No special brackets, just clean "$myvar"

# Reading files is easier

Julia

    readlines("my_test.txt")

open("my_test.txt").readlines()

What a stupid option in 2? if I call readlines on a filename, 99% of people, 99% of the time, want to read the lines on the file at that path. Why require two function calls?

I responded in detail him about, but honestly I can't think of many that answer your qiestion. It's amost always the opposite. A I like to present this as stereotypical example of the types of differences you find the two, IMO. Generating a 3x3 matrix of random booleans.

>Julia

  A = rand(Bool,3,3)

>Python

   import numpy.numpy as np
      gen = np.random.default.default_rng()
      B = gen.choice([True,False],(3,3))

I think they just mean that with trivial projects, it's not worth trying them in a new language since the performance benefits are probably going to be minimal, and wouldn't really show off Julia's strengths.

I both agree and disagree. It does look weird as library vs language. However at the same time, in Pandas everything tends to be in the Pandas library, whereas when using DataFrames.jl you tend to mix it with a lot of features that are external. Most of the calls just use overloaded functions from Julia's Base library (mean, first, last, findall). The Pandas model is to look at the Pandas docs and find the Pandas dataframe function that does your job. The DataFrames.jl model is to do whatever you would have done normally in Julia, like use the sort function, but now just use it on a DataFrame. The idea of DataFrames.jl is that you know the language and so it extends/adds as few functions as possible (joins, groupby, split-apply-combine I think are it?). This plus many other calls use functions from the more general Julia data science ecosystem (CSV.jl, JSON.jl, ...). So the title ends up being a bit apples and oranges, but the usage is also quite apples and oranges and the cheat sheet does accurately reflect that.

You have many errors in the tables, for example the pandas indexing is obviously wrong.

Ok, in that case Python would be 0, 1, 3 and Julia 1, 2, 3. My point is that the example explicitly skips an index in the definition of a data frame for Python, but it doesn't for Julia.

This makes perfect sense from the perspective of a language designer/computer scientist who is trying to keep their design clean and consistent.

It makes no sense to an end user that expects an argument you pass to BigFloat to be treated as a BigFloat. As an end-user I would rather have a warning or even error than to have my argument silently treated as a double.

You can just use Polars[0] instead of Pandas and easily beat both Pandas and DataFrames.jl

Pure Julia is faster than pure python, but there are non-pure python tools available in the python ecosystem for a ton of things.

[0] -- https://www.pola.rs/

Look up the termux community. There are actually people in third world countries who are learning programming using their phones. I've seen awesome builds which use Android phones as their primary CPU unit and cobble on scavenged monitors and keyboards and mice. It's honestly fucking awesome

How’s that related to the article not being mobile friendly?

We should nurture more accessibility, in this case, mobile compatibility.

For instance, consider someone who has limited access to desktop computers and have to go by with a mobile device. These individuals do exist, and their access is as legitimate as any other.

I guess you could find the cheat sheet useful on a phone/tablet while you are learning the correspondences.

[dead]

ibis in python is a really nice middle ground. Nice API + in a programming language, but executes on database backends (which could be polars or duckdb on in memory arrow tables).

  mydf = pd.DataFrame({'a' : [1, 2, 3]})
  print(duckdb.query("SELECT SUM(a) FROM mydf").to_df())

> The SQL table name mydf is interpreted as the local Python variable mydf [...] Not only is this process painless, it is highly efficient.

It might be painless and convenient at first, but I feel like this could get you in trouble down the line. Is there a way to avoid this?

Duckdb is sick. You can also do queries on parquet, etc.

I highly doubt that SQLite is faster than pandas, let alone dataframes.jl, for analytical workloads.

SQLite is good for a bunch of stuff, but it's terrible for analytic workloads. Not even in the same ballpark for pandas, let alone Julia

A lot of data scientists don't know SQL and don't understand why people use it. That said, there are cases where in memory workloads and certain manipulations are less encumbered by dataframes APIs. But in a lot of cases... They get abused by people who really do need to push themselves a little bit to learn something new.

You use both. Once your data fits comfortably in memory it's naive to try to build histograms, pivots and charts using pure SQL.

You're saying all Pandas usage (an incredibly popular library) is because people don't want to use SQL?