
A JSON field type for Django - aychedee
https://metric.io/blog/2014/07/json-field-type-django
======
kanja
Hopefully this will soon be added to django core - it's one of the goals as
part of Marc Tamlyn's kickstarter to improve postgres support in django.

[https://www.kickstarter.com/projects/mjtamlyn/improved-
postg...](https://www.kickstarter.com/projects/mjtamlyn/improved-postgresql-
support-in-django)

------
jessedhillon
Really not trying to troll here, but I want to know why anyone uses a Python
ORM here other than SQLAlchemy? It's been a while since I used Django's ORM
but I recall the comparison between the two being very heavily in sqla's
favor.

~~~
grantcox
I'm fairly new to Python (~6 months) and started with SQLAlchemy, but recently
switched our app to Peewee
([http://peewee.readthedocs.org](http://peewee.readthedocs.org)) because of
frustrations with the SQLAlchemy "session".

The standard method of scoped_session(sessionmaker(engine)) will return a
thread-local session - so all data manipulation effectively shares a global
session.

\- Every change needs to be followed with a commit() or rollback(), or you'll
end up with rubbish in the session that some later call will inadvertantly
commit.

\- If you want to do a general "update where" call, make sure you use
"synchronize_session=False" and then session.expire_all(), otherwise the
session will be out of date.

It felt like every time I had code working with data, I also had a non-trivial
amount of session management. It wasn't abstracting the database access away,
it was making everything very SQLAlchemy session specific.

To be honest I'm somewhat second guessing my decision, because Peewee is far
from the Python standard. But it's simple, understandable and clear, and
doesn't force a strange "don't forget to manage the global state" mindset onto
everything.

~~~
jessedhillon
FWIW I use Pyramid with SQLAlchemy and it comes with a transaction manager
plugin which automates the begin, commit and rollback on error of
transactions. 99% of the time I don't need to think about transactions, only
during CLI tool development and only barely. YMMV

~~~
aychedee
Yeah, sounds like you're mileage does vary.

------
tonylampada
Have you looked at [https://pypi.python.org/pypi/django-
jsonfield/](https://pypi.python.org/pypi/django-jsonfield/) before building
your json field? If so, can you elaborate about how those implementations
compare with each other?

~~~
tobych
My understanding is that the field, when working with a PostgreSQL database,
is stored as a JSON field in PostgreSQL, rather than just a text field. I
don't think django-jsonfield can do that.

~~~
cpbotha
There are a number of existing jsonfield implementations. This one, probably
the best of the lot, DOES make use of the PostgreSQL JSON field:
[https://github.com/bradjasper/django-
jsonfield/blob/master/j...](https://github.com/bradjasper/django-
jsonfield/blob/master/jsonfield/fields.py#L136)

It looks like your app does more in terms of casting data that comes from the
field. Is this the major improvement?

------
geweke
It's not Django, but I've built something quite similar for Rails, and have
really loved using it. The combination of RDBMS stability and the flexibility
of schemaless content is really powerful -- it is (IMHO) the best of both
worlds.

Would love to hear people's experience with this kind of setup (no matter what
software you're using)...I think it's a very interesting, and useful,
direction to go, especially for new projects where you may not want the
overhead of multiple storage systems.

[https://github.com/ageweke/flex_columns](https://github.com/ageweke/flex_columns)

------
damon_c
Thanks for this! I use some of the various json fields that exist already but
have been feeling like it is about time to start using the built in Postgres
JSON support instead of just TextFields.

------
jpdlla
This looks great, but how does it compare to already existing options like
django-jsonfield or jsonfield?

~~~
aychedee
I looked at lots of alternatives. But I needed something that supported the
actual underlying Postgresql json type. Psycopg2 converts those automatically
into Python types. So we needed a JSON type that would accept: Python dicts,
lists, strings, and JSON encoded objects, lists, and strings. None of them
support that because they are all just storing the data as text in the
backend.

~~~
Jasber
I run django-jsonfield which has partial support for native Posgresql JSON
type and more will be added as it's incorporated into Django:
[https://github.com/bradjasper/django-
jsonfield/issues/55](https://github.com/bradjasper/django-jsonfield/issues/55)

Happy to accept pull requests for stuff like this if you're interested in
contributing.

~~~
aychedee
Great, I'll definitely do that. Maybe we should have a DM twitter chat on how
to generalise it before I submit anything? I'm hanseldunlop there.

------
andybak
Remember to add it here when it's released:
[https://www.djangopackages.com/grids/g/json-
fields/](https://www.djangopackages.com/grids/g/json-fields/)

------
Pitarou
What are the engineering trade-offs here? Metric were migrating from CouchDB,
so using a JSON field is the obvious solution, but what about for new
projects?

Compare it to say, Django-CMS, which achieves pretty much the same thing by
storing the tree data-structure directly in SQL. The `django-mptt` library
handles the details of efficiently implementing a tree structure in SQL, and
`south` simplifies database scheme migration. As a programmer, which system
would you rather work with?

~~~
aychedee
For myself? I don't want to do a migration whenever we add a new widget to our
promotion or email builder, I certainly don't want to think about doing a data
migration on every existing document. In this particular case we do not care
about the contents of this field. There is already a lot of related
information stored in the same row.

------
misiti3780
You should use hstore to store json in postgres also - no?

[https://github.com/djangonauts/django-
hstore](https://github.com/djangonauts/django-hstore)

I think the issue with that is it converts everything to string so you need to
use json.loads().

~~~
petepete
Not quite, hstore doesn't support nesting.

------
dangayle
You should also get this up on Github quick :)

~~~
aychedee
It's probably more appropriate to submit a pull to one of the existing
projects TBH.

