Well, I enjoyed your book, so I’m interested in learning more about this from you. Are you asserting that complicated (perhaps not the best choice of word in these “complected” times) Clojure (or Lisp) leads to this, or that this is inevitable and all Clojure (or Lisp) programs that happen to be “complicated” are destined to reimplement half of Haskell. If you mean the latter, do you mean to suggest that Lisps are not the way to go if complexity is unavoidable?
Not at all – I do think Lisps (and Clojure in specific) is very much the way to go to build the complex systems required by today’s world. However, once you go beyond a certain point, the lack of a type system really starts to get in the way of productivity and quality.
I predict people are soon going to build their core systems in Haskell and use Clojure for all the stuff around it – DSLs, web stuff perhaps, other tooling.
>> Any sufficiently complicated Clojure (or Lisp) program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Haskell.
In principle, I disagree – please let me defend my opinion. My purpose is not to champion Clojure / dynamic typing but simply to explore your very thought-provoking Rule (expressed in other words as well by other programmers, from time to time). I’d be happy to get some of my ideas fixed if they turned out to be mistaken.
The first thing that calls my attention is that you treat Clojure and ‘Lisp’ as the same thing. Clojure is freed from much of the ‘Lisp Curse’ described by the previous commenter: immutability, laziness and the FP approach in general *are* the default in Clojure (and, obviously, are implemented already). Whereas traditional lisps don’t even have a partition function (see Xah Lee’s comment at https://groups.google.com/forum/?fromgroups#!topic/gnu.emacs.help/_p2-GXAANgw ).
So when you compare Haskell with Clojure – there’s only one step of difference -overgeneralizing-: typing. If we talk about Common Lisp, sure, we’ll end up reimplementing a whole lot of constructs, and the Rule would be accurate.
Additionally, I find no good reason to compare Haskell *precisely* with Clojure and lisps: if Haskell is better, it is better than anything else. So a better formulation would be:
>> Any sufficiently complicated program not written in Haskell contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Haskell.
Which we could reduce to “All substantial programs should be written in Haskell”. If this was true, it would be *obviously* true, however it isn’t. The language has had plenty of opportunities over the years and it just hasn’t taken over. Will it ever do?
One thing that negatively calls my attention about Haskell is that I’ve found no compelling reason to use it. There’s the recurring argument of the “massive systems” which one can only deal with with the help of a static type system. But *who* wants to write such a system? Do there even exist systems whose nature is so interlocking that you can’t decompose it in libraries, and abstractions in general? In other words, how many lines of code do you really need to describe business logic? They should belong at the very most to the order of thousands, not millions.
I believe there’s no such thing as an intrinsically ‘million-LOC’ project and that if you produced one, you’re doing cognition wrong. On the other hand, Clojure-style functional/dynamic programming does not particularily alleviate this problem. But neither does a sofisticated typing system! Declarativeness -the ultimate simplicity- does – which you can do in both Clojure and Haskell.
Leaving the ‘incomprehensible systems’ argument aside, what are some good reasons to use Haskell’s set of abstractions and constraints? I honestly haven’t found a single one – all I can encounter is monad tutorials, or at best, brief, unjustified enumerations of its supposed benefits. Whereas Lisp advocates such as Paul and Rich have made some incredibly appealing points.
Ultimately, your Rule is nothing but the old ‘static > dynamic’ claim. I’ve already visited the subject of static typing. As for its dynamic counterpart, its advocates often argue in favor of ‘ad-hoc static typing’: duck typing, monkey patching. I find this is actually worse than plain static typing: the semantics are equal, but the rules are just convention, hence easy to forget.
The ideas mantained by Stuart Halloway in ‘a Java Better than Java’ or ‘Evident Code, at Scale’ hint us how a better dynamic data modeling might be: generic data manipulation would be the ideal – no DSLs. I won’t repeat his reasonings here as you’re probably familiar with them.
At the end of the day, programming has a lot of room for subjectity: there is more than one way to do it. One can develop substantial, correct systems with either dynamic or static languages without drowning in complexity; other factors -which currently are pretty underrated- matter much more.
Yea – in the end, my comment is just about static typing. And I believe in most non-trivial applications, eventually, the lack of a type system becomes a problem. I love Clojure (obviously), but I care more about successful systems (in all senses of that word – quality, productivity, speed of development (in the long term), robustness, performance, etc). And I think language-level support for strong (and static) typing can help these goals. I’ve certainly felt the lack of it with Clojure, especially when you scale your team size.
Perhaps Haskell isn’t the right solution to this – and it’s something else. I’m fine with that – I only care about the outcome. And I’d still use Clojure liberally.
I’d be interested in hearing a more nuanced discussion about this Certainly it’s obnoxious when simple type errors get into production and Clojure doesn’t offer much there. However I think just as many errors occur from subtle invariants where the types works out just fine. In your projects, do you find that the former outnumbers the later?
I don’t have enough experience with Haskell (and certainly not with large systems that use it) to throw much light on this, beyond the fact that Clojure development always seems to get to a point where it needs something more. Perhaps typed Clojure will provide a path forward. Or maybe it’s just something I haven’t gotten an understanding of yet…
I think both languages are powerful enough, and different enough, to say that about each other. As evidence of Haskell’s partial implementation of Lisp, I present Template Haskell, which for example is used extensively in Yesod.
Amit: Your post is interesting, and I agree with your statement about using Clojure and Haskell for a large complicated project. I also took your statement to mean it is OK to use multiple languages to solve a problem and the language or languages best suited to solving the problem.
For this upcoming December, I am working on automating a big real-estate data transfer (for tax bills, of course) that used to require three separate files, each of which had to be hand-massaged in Excel. The language I plan to use to do this is older, but well suited to the job, Informix 4GL. This doesn’t mean Clojure or even Haskell, if I knew it, shouldn’t be used, but my feeling is why not use the native language that is better suited for my task.
So, if strong typing is important, using Haskell makes sense. However, what about Scala? I am curious as to why you would not turn to Scala.
I would really like to know if you could give an short code extract with context of “any sufficiently complicated Clojure”?
I think that Víctor M. Valenzuela made a very good point with:
“In other words, how many lines of code do you really need to describe business logic? They should belong at the very most to the order of thousands, not millions.”
But obviously your coding expericences must have been different?
I am also surprised that you are missing a type system so much because I know you write test cases
for every function you develop and would have thought that this would have helped you alot with missing types?
What do you think about using a clojure database like Datomic? Besides from being an example of a “complicated clojure” programm that seems to work I had so far the experience that it also gave me
for my “business logic” just enough type safety as a consequence of its schema
It’s not about the number of lines, quite honestly. In the beginning, when the team is small and you’re following all good development practices (TDD etc), you’re fine. As the code-base expands and more importantly, the team grows, sometimes it becomes difficult to keep up the quality. It could be some developers not writing tests carefully, or some missing knowledge.
My theory is that type-systems would help. Just like Clojure’s built in concurrency and state-management semantics *ensure* that there are no threading related bugs. Automatic, language-level enforcement goes a long way.
In my experience, static typing helps the quality start off low at the beginning, and proceed to decline from there. I’ve especially withenessed this over the past several years working on C# teams. Most everyone leans against the compiler, because we all tend towards laziness. If I had a nickel for everytime somebody told me that they didn’t need to write unit tests (much less use TDD) because the compiler would catch “breaking code”…well I’d have less than a dollar, but the attitude is prevalent.
I think you are right that *eventually* you could find some benefit from static typing, but I think your system likely will be of much better quality if built up in a dynamic language (with the attendant discipline).
To be clear, I don’t have any bone to pick with statically typed languages (I prefer Lisp, Smalltalk, etc., but C#, Java, and C++ have almost always paid the bills.) – high quality *can* be achieved with them; it is just not likely.
Nothing can replace really good people, and you need the discipline and practices to stay in place. And when I think static-typing, I don’t think Java (or its cousins). Just as I don’t think of those languages when I say “OO”.
Advanced type systems (with inference) are different, and come with some advantages which can help even when you have the best people and the strictest of practices. For example, wouldn’t you use a typed version of Clojure? Being able to use the right language/tool for each part of a system is important – and the notion I’m exploring is just a natural extension of the polyglot system idea. I guess I’ll have more to say once I actually build a system in Haskell + Clojure.
Looking forward to learning what you discover from a Haskel+Clojure system. I do see your point about advanced type systems and thier attendant advantages
Unfortunately my experience with polyglot systems has been on the order of PHP front-ends (ahem…) and .NET “services” and a mess of legacy SQL thrown in for good measure.
I differ on what Clojure’s approach to change is all about. I’d say it’s not about enforcement or guarantees, it offers instead higher-level abstractions. Nothing *stops* you from creating unsafe code.
You probably know better than me that functional-style concurrency doesn’t entirely alleviate “threading related bugs”. It introduces its own set of issues, *but* those issues will be substantially simpler to reason about: they’ll mostly relate to how you model your data and formulate your functions rather than, say, lock acquisition order. This relates closely to my point.
>> As the code-base expands and more importantly, the team grows, sometimes it becomes difficult to keep up the quality.
One could hardly disagree with this. But, as I mantain, typing isn’t the main issue here. Our usual divide-and-conquer approach would be the culprit instead. Divide-and-conquer (‘architecturally’ speaking, not algorithmically) is sure a great general principle, but past a certain point, it won’t scale. Please let me justify this, if it doesn’t sound evident enough already.
Say that the maximum amount of knowledge a programmer can reasonably hold about a system is N (N not necessarily measuring LOC). But, under a traditional approach, the project will have, for instance, a 10N size (12N if you’re using Java, 8N if you’re using Haskell – it doesn’t matter). So basically no one understands the system but at a high-to-medium level. As you add more programmers, the communication overhead grows exponentially.
The problem I describe is not new at all, and neither would be the nature of its possible solutions: less how / more what, and bring the control of the business flow back to the user. These features may still look like just illusions (there are some success stories, though), but we still are at the early days of computing – who knows how unsofisticated our practices might look like in a future!
One read that comes to mind is Steve Yegge’s “The Pinocchio Problem” which spans both the topics of large systems and typing.