Shane Hastie: Hey folks, it’s Shane Hastie here. Before we start today’s podcast, I wanted to tell you about QCon London 2024, our flagship international software development conference that takes place in the heart of London next April 8-10. Learn about senior practitioners’ experiences and explore their points of view on emerging trends and best practices across topics like software architecture, generative AI, platform engineering, observability, and secure our software supply chains. Discover what your peers have learned, explore the techniques they are using, and learn about the pitfalls to avoid. Learn more at qconlondon.com. We hope to see you there.
Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today I’m sitting down well hours apart on the opposite side of the world with Tim Ottinger. Tim, welcome. Thanks for taking the time to talk to us today.
Introductions [01:05]
Tim Ottinger: Shane, it’s so good to see you again. I look forward to every chance we get to talk.
Shane Hastie: I appreciate it. So you are with Industrial Logic. You’ve been with Industrial Logic for a fairly while?
Tim Ottinger: Yeah, 13 years, if you can believe it. I’ve never been anywhere 13 years in a row.
Shane Hastie: So there’s obviously something special about that place. But before we go there, tell me a little bit about who’s Tim?
Tim Ottinger: Well, hi, I’m Tim. I’ve been programming since 1979 with a job I got accidentally and I had suddenly fell in love with programming. I got a call one day, so I get a call this day. This guy calls me up, he says, “Is this Tim?” and I said, “Yes.” And he says, “Oh, okay, Tim, this is Ed Arthur. Do you know me?” And I’m like, “Oh yeah, long time ago.” He says, “Yeah, I figure you’re probably about 16 by now, aren’t you?” And I said, “Yes.” And he says, “Do you have a job?” I said, “No.” He says, “Do you want one?”
“Of course.”
He says, “Okay, grab your Social Security card, come on down.” And I said, “Well, I got to get my Social Security card.” This is a long time ago. And he says, “Well, when you get it, pop on in. I got a job for you.” I said, “Okay, well Ed, what will I be doing?” He said, “You’ll be programming computers.” I said, “I have no idea how to do that.” And he goes, “Nobody does.” Remember, it’s 1979. And he says, “But we’ll teach you how.” And I’m like, “Oh my gosh, what happens if I can’t do it?” He goes, “Well then, you’ll quit or we’ll fire you. But until then you get paid to learn.” And we struck a deal.
Turns out this guy had played piano at a church I went to when I was a kid, and they had a real simple job and they were hoping to get some high school kid to come in and do it cheap. And I became the high school kid and I was able to do it cheap. So I got into it, learned to program, and I’ve hardly done anything since. I’ve been in software I think one summer I worked at a warehouse because there was a girl. And other than that, this has been my whole life. I worked with Object Mentor during the days when OO was the new idea until well into XP in code, and again, Industrial Logic for 13 amazing years.
Shane Hastie: So what’s special about Industrial Logic?
Technical excellence at Industrial Logic [03:09]
Tim Ottinger: Well, it’s kind of a window into the future. The way we work is far enough advanced that sometimes it’s startling for people, but it’s also, we’ve got good camaraderie. I’ve got an amazing team of people. I wish I could keep them all with me all the time and work with them every day. But we are all coaches and mentors and trainers of some experience and some background. So it’s easy to trust people and to work with them. We fall into a rhythm easily. And our clients are surprisingly large and surprisingly multinational, so there’s always a lot of exciting travel.
Shane Hastie : What are the technical practices that Industrial Logic brings to the table when you’re working in these clients sites?
Tim Ottinger: Somebody said that watching us is like seeing aliens at work. It’s a little different. We are a continuous deployment shop. We believe very much in doing things in a very incremental and iterative way. Josh is an XP-er way back. I’m an XP person from way back. So there’s a lot of TDD. There’s programming in pairs and teams. So ensemble programming, which we used to call mob programming, but rightfully don’t use that term anymore. Working in groups of people with different talents and skills. Slicing our work very, very thin so we can deliver many times a day with confidence with tests. We work so collaboratively that that’s unusual for people. A lot of people are used to having solo assignments. You only get together to divide the work up and you only divide it up to assign it. And then you get back together when all the work is done to try to piece all the junk back together. We skip that whole thing. We’d make everything work all the time.
So micro testing, micro commits and teams, together I think, is such a unique combination compared to what most organizations are doing that it gives us a significant speed advantage. And also it allows us to adjust and integrate well. So that’s a lot of the magic that we bring to it. And I know that that may sound really bizarre, but also remember that Netflix years ago was committing to production 300 times a day. So this is not something unprecedented and bizarre. It’s just skills. Like everything else, you learn how to do it and you do it.
Shane Hastie: Let’s dig into that ensemble programming in particular. It’s gone up and down in terms of flavor of the month or flavor of the year I would say. When Woody Zuill and others were publicizing it in the first place when it was being called mob programming, there was quite a lot of publicity and take up and then it seems to have lost some of that edge or has it. Or has it just become mainstream?
Tim Ottinger: It is been almost 15 years, hasn’t it?
Shane Hastie: Yeah.
Ensemble programming [05:58]
Tim Ottinger: Wow. That’s hard for me to… I’m sitting in my head doing the math. It’s like 13 to 15 years old now. And of course when they started talking about it, it was such a radically upsetting idea that when people heard it, they either were immediately polarized to, “Oh, I’ve got to do this,” or, “Oh no, we will never do that. It’s such a stupid idea. We shall never do it.”
I think that the shock has gone out of it. So that initial ripple from this is a mad, crazy idea is gone. And now more people, I think, can quietly think about it to not feel like they’re somehow violating corporate standards or anything. They don’t feel guilty, but I don’t think it’s become nearly mainstream enough. I’m still finding that most places, they’re isolating people. And especially post-COVID, right? A lot of people think if you’re work from home, you must be work alone. And that’s not what we do. We work remotely in teams. We work remotely in ensembles.
But I think that the idea that people are alone and they must be treated as alone, and of course the old remaining belief that the best software is written by one person sitting in a dark room all by themselves thinking really hard, which doesn’t seem to be borne out by reality, those things hold onto people. So people are thinking, “Could we put people together? Is this a thing that can happen?”
So I’m finding a lot of companies now, I’m talking to people and they’ve started with pair programming and they’re moving to ensemble programming, which by the way is probably backward. And they’re doing it quietly within their company and nobody’s raising a fuss and there’s plenty of material out they can read about how to do it well. So I think it’s growing still. I think it’s just in that steady growth. It’s no longer a radical scary idea that has to either be brilliant or moronic. It’s no longer scary, but now it’s just down to how quickly people are assimilating the ideas. And of course, whether they have a crisis is large enough to make them want to change how they do their work.
I know that’s an odd statement, but most people are at a steady state and it feels like it’s working well enough. It’s efficient enough that they’re not bleeding money through development too bad. It’s within the margins. They’re successful, so therefore, whatever they’re doing must be the right thing. And it’s usually at some point of crisis, they’ve suddenly lost people or they’ve got a financial crunch or they’ve had a project go particularly badly, then they think, “Well, maybe the way we’re doing this is not perfect, and maybe we should consider some other way.” I believe that this ensemble program probably spreads by crisis primarily up until probably five or six years ago, and now there’s a steady quiet adoption.
Shane Hastie: What is a good ensemble programming session feel like, look like?
Describing an effective ensemble programming session [08:49]
Tim Ottinger: So let’s say that we have a new piece of work to do, we kind of know what we’re going to do. Maybe we have a UX person, a UI person, a back-end front-end coder, maybe a tester in our group. So we have some number of people, probably less than 10 typically. We take a look at the problem and we decide how we’re going to tackle the problem. We kind of make some back-of-the-napkin sketches, and we fire open the code editor. So once it’s open, we’d make sure that we have a clean start. There’s no leftovers, uncommitted files, no junk. All the tests are running clean. We’re current with the mainline. Great.
Given a clean start, now we start incrementally building the system. We take a little piece of it front-to-back, a little front-end, a little back-end, a little middleware, a little database, some tests as we go. So we’re doing a TDD. Generally, the test’s written just before the code that needs to pass the test. And when the tests run green, we commit. Git is wonderful because it lets us commit locally, so I can always save my game before I do the next tricky part of the work. So together we can focus on what we’re doing now. And then if the next piece doesn’t go well, we can always back up. So there’s a ratchet. We safely move from step to step. Maybe the second or third step we decide to integrate, so we pull from the mainline again together. We look for integration problems, merge conflicts. If there aren’t any, the tests run green. We push live.
And we’ll do this. We’ll go live in a mob maybe seven or eight times, maybe 12. And of course, you don’t go live without pulling the new code and checking the integration. So it’s nice. And we’ll probably take breaks maybe every hour, five to 10 minute break every hour. And of course a nice long break for lunch. And we’ll probably do this kind of work from like 9:00 AM till maybe 5:00 or 6:00 PM and then call it a day.
Shane Hastie: Who’s holding the keyboard and what else is going on?
Tim Ottinger: When we were live together pre-COVID days, we used to actually have a keyboard set out and a big screen and we would have Clorox wipes, some kind of an antibacterial cleaning solution. One person would move up to the keyboard and you’d work for somewhere between five and 15 minutes at the keyboard. That’s how you take a break, by the way, in an ensemble. The one typing is the only one not programming. You’re just operating the keyboard. The rest of the group is deciding what to do next and they’re spotting problems and they’re suggesting new tests. They’re thinking ahead. You’re just taking a break and typing. And they say, “Well, we need to make a new class, class this. Off you go.”
So you rotate out because you get bored being the typist for the group and you want to really be doing the thinking and somebody else moves in, which is really nice. Sometimes we bring in a CEO or a COO and they’ll join us and they’ll take their turn at the keyboard and we’ll explain to them, “Oh, you need to do the option command L and do this.” And then when they rotate out, the statement is, “Oh, we need to extract that method and rename it to this.” The granularity of direction changes with the person at the keyboard. But a lot of what’s going on is that the people who aren’t typing are thinking. And programming is easily 11, 12 thinking. So it actually makes sense to work this way.
Now, we’re no longer in the before time we’re post-COVID, and that makes it a little bit more interesting because we’re all on a Zoom call. Well, how do you change drivers now? Well, the code currently is on my machine. I commit. If I push, then you can pull on your side and you can drive for the next few minutes. Well, now five minutes is really hard to do when you’re remote. But 15 to 20 minutes is kind of long. So we try to shoot somewhere in that 10, 15 minute window and we just take turns that way. Get it to be green, get it to be safe. Always work in safe steps that you can push. Never have something half-written and not running and not good and visible to the customer all at the same time. Always keep it running. And that’s really what we’re doing, is figuring out what the next slice is, how to know that this slice is good and how to trade easily and quickly.
Shane Hastie: What if we fundamentally disagree about the approach that’s being taken?
Dealing with disagreements [13:03]
Tim Ottinger: Oh, we do that all the time. That’s perfectly fine. It’s kind of funny because people wanted to argue then on principle. They want to back up and get to these abstract principles about, “Well, readability is really important.”
“Oh, but this other thing is more important than readability.”
“Well, no, maintainability is based on readability. Blah, blah, blah.” It turns out that once you’re down into the cases and you got the code in front of you, a lot of that vanishes. But we still think about the architectural side of things. What’s the real big design here? What’s the big architecture? And as we do that, then the question is, “Okay, well maybe it needs to be this shape, maybe it needs to be that shape. How could we tell?” And this is a key part, is you’d make your decision when the decision becomes clear. So when we disagree, it’s like, “Oh, that’s two paths or three paths we might go down. How will we decide? What’s the indicator?”
“Well, I think the next thing that’s going to happen is this, and that would indicate we need to go this way.”
“Fine. Let’s keep going for a little longer. Let’s talk about it in a half an hour.” I think that that’s really helpful, that the idea that just because an idea is in my head, we don’t have to act on it yet. Let’s collect our evidence and make a choice.
Shane Hastie: You mentioned the cross-functionality and the different skill sets that are there. Is there a natural relationship to the person doing the typing to the skill set that is needed at the moment?
The inverse relationship between skillset and driving in an ensemble [14:28]
Tim Ottinger: No. Usually an inverse, if anything. And usually you’ll have more than one programmer, a code generating human being in the group. So you probably have two people who know enough about the coding to help direct it and move forward. And of course, if you’ve been doing this for a while, everybody’s seen everything. So I would think that if it’s a UI issue that you’re working on, it’s nice that the UI person is behind the typist somewhere suggesting a new layout, maybe doing a sketch rather than actually being the person who’s keying in the JavaScript for the component.
Shane Hastie: What are the anti-patterns? How can this go wrong?
Anti-patterns and pitfalls [15:06]
If you do it well, then you’re doing a three person or a five person job, and it’s sort of like you’re focusing all the talents on this thing that needs to be done. If on the other hand, we get together and we break down the work according to a person’s individual skills, so I happen to know that you’re a really great React to JavaScript programmer, so we’re going to carve off the React bits and I’m going to assign your name to that, and then we’re all going to work on it. Well, I don’t know. It’s like having four people playing the same flute. It feels really extraneous and artificial. The work is carved for one person and four people are working on it.
It actually doesn’t make sense to have four people do one person’s job. Four people should do a four person job. And if we over divide the work, it becomes moronic really and you end up with three people watching one person work. That’s not good. That’s not an efficient use of human IQs and experiences. How do I apply that when it’s specifically written to exclude me?, right?
So that’s one of the ways it goes wrong. And the other one is you get stories about surgical teams and pilot and co-pilot relationships where the lower ranking person was not to challenge the higher ranking person, right? If you get into a group programming, whether that’s pair or ensemble or micro swarm, mini mob, whatever, and if I’m not really allowed to challenge what you’re doing or to second guess you or to think around you, then we end up with puppeteering. I end up to be just a pair of hands for your brain, which by the way is inefficient use of resources. You could use your own hands faster than you can use mine. And also it tends to engender resentment. Most people don’t want to be somebody else’s robot. We like to engage and learn together.
Another one is that people don’t want to waste time learning new things in an ensemble. “Well, we need to just do this. We need to be hacking code the whole time. We don’t want to waste time looking things up or figuring things out.” Well, that’s exactly wrong. The whole point of the ensemble is to focus multiple people’s knowledge on one problem. It’s a big magnifying glass. You want to bring all of that sunlight down to one point. And that’s our job, to bring light to the work we’re doing.
Shane Hastie: When we were chatting earlier, you made a statement that I found intriguing, and I’d love to go deeper, the issue or the challenge of code readability. And you made the statement, “What we thought was always wrong.”
What we thought about code readability was wrong [17:35]
Tim Ottinger: Yeah, we can go back a long time. I think my first splash in the world was on coding standards and code reviews way, way back in the ’80s. So I’ve got a lot of history with this. We thought that readability was an attribute of the code we write, right? You have this artifact, this file, and it’s readable or it’s not readable. There’s a sliding scale. “It’s somewhat readable. It’s a lot more readable. It’s very readable. It’s nearly impenetrable.” That’s the code. And so we were focused on making that artifact be just right. It took a long time.
There’s a strange thing we would do. We would look at a way of writing, expressing a code idea, and we’d say, “Oh, that’s more readable.” And we’d show it to other people and they’d go, “Oh, what’s that? That’s kind of strange and unfamiliar.” And we’d go, “But yeah, but look how readable it is. So go read it.” And they would start reading it, and then they’d go, “Oh yeah, that’s readable. That’s really nice. I like that.” And they’d show it to somebody else, “Oh, what the heck is that? That’s so weird. Oh, I get what you’re doing. That’s very readable.”
So we generated readability accidentally because what we didn’t realize is that readability isn’t an attribute of the artifact. I don’t read Norwegian street signs very well. I certainly don’t read Chinese street signs very well. I do not find them readable. However, if you’re Norwegian, the Norwegian signs are perfectly clear. And if you’re Chinese, the Chinese signs are perfectly clear. Why is it readable to one person but not another? Well, the reason is because it’s an attribute on the relationship between the two. Readability is an attribute of the relationship between the artifact and the audience. Not even the author, but the audience. So the code that’s readable in your company might not be the same code that’s readable in my company. It might not be what I would want to publish in my book, but it might be what you’d publish in your book. But people would like them both ways. Different people are going to see it. There’s a bit of subjectivity.
If I back up a little bit, when we were teaching code readability, we were actually adjusting the audience to the code that our audience likes. We were bringing them into the same audience. So our audience appreciates this way of writing code. I’m going to teach you how to read code that’s written this way, and now you’re going to appreciate that. You’ve become part of the audience that I’m building. We probably could have chosen any code patterns. They probably would’ve come out more “readable,” in air quotes, because we taught the audience to expect and appreciate it that way. And we didn’t know. We thought we were doing objectively good work on a subjective topic.
But when you know that, it’s really cool. So you go into your company and the code is horrible you think, and you think, “Oh wait, I’m not the audience that this was written to. Maybe I should learn to read this code because I could learn things that will make this code more obvious to me, but I’ll also bring my history to it” and I’ll see the code and I’ll say, “Oh, we could express this idea this other way,” and that audience might adjust to some of the patterns that we’ve learned to appreciate. We’re a blending of styles. So when I move in, I bring my barbecue skills into the Scottish community, they’re going to be like, “Oh, now I like barbecue too. It’s different from Scotch pie, but I like it also.” And so it’s a blending of things as we come together.
Shane Hastie: But barbecued haggis is not going to be a good thing.
Tim Ottinger: I don’t think that’s going to sell very well. It’s kind of hard to keep it together on the grill too. Can you see how that changes our approach? Now we’re maybe thinking about educating the audience. But also the author isn’t the audience. And when people say, “Oh, code style is completely subjective. That’s not a tenable problem. You can’t fix that.” It’s like, “Well, if it’s strictly between the author and the artifact, then it is purely subjective and there’s nothing we can do. I mean, it’s within a band. You might like vinegar, potato chips, plain potato chips or barbecue potato chips, but you probably don’t prefer paint chips and wood chips, right? I mean, there’s a band of subjectivity in a broader band of universality. But if it was just you and your code, then how you write your code is how you like your code, and that’s all there is to it. But you’re only one member of your audience.
If you’re not writing code strictly for yourself, then now we know that readability is us writing for our audience, the other people who are going to be in our code base, including future us. The Tim that comes in a year from now isn’t the same Tim that came in last year. And so now we have to think about how do I adjust for my audience? How does my broader… It’s like publishing a book. If you write a book for PhDs or you write a book for business people, it’s a very, very different approach. Business people are in a hurry. PhDs are academics. They’ve become accustomed to really having to dig to pull out all the meaning from the text. Business people don’t have time for that. So how’s your code written? Is it written to be extremely accessible by people in a hurry or is it written to be difficult and dense? Maybe both are right.
Shane Hastie: Can I ask you to peer into the crystal ball for us? We’re in an interesting transition and an interesting state, I would say, in our industry at the moment with generative AI and Copilot and tools that are doing some interesting things. But where are we going? What is the future of programming? And I won’t hold you to this.
Opportunities with AI [23:01]
Tim Ottinger: Well, it’s always hard to think humanity because we could be spending so much time and medical knowledge solving deep cancers and things, but a lot of our money goes into solving male pattern baldness. And you think, why is that the thing? Why do we have such trivialities that are dominating? Well, because that’s where the money is. So what people will buy will probably gauge where AI is really going to go.
What I’d like it to do, let’s stay with software for a moment, what percentage of the corporate code base does the average programmer have committed to memory? I’m what? Intimately familiar with a small percentage of my total code base. Well, what an opportunity, right? What if I could be augmented with systems that look at the whole overall code base, even mechanistically? And they know, “Oh, Tim, you’re editing here. These other 18 places that use that code have these tests. And by the way, this is going to affect a database, right? That’s going to affect these reports. Can I scope your work for you? Can I help you understand the broader context of the thing you’re doing?” And if it’s only like 85% correct or even less than that, it can still give me great insight into what I’m doing. It can augment my abilities and even five of us together, that each know 1% of the code base, that’s really not the kind of coverage you’re looking for, right?
But also we’ve already seen with our editing tools, how much auto-completion and suggestion can help out. Again, maybe we’re using five programming languages every day. Okay, what percentage of the grammar of all of those programming languages do I really have committed to memory? Do I really know 100% of all the grammars of all the languages? Probably not. I probably would be stretching it to say that I really know 60% of everything that’s in JavaScript and the 85 libraries and it’s standard library.
Python. I’m pretty good with Python. I know most of Python I think. But do you know there’s over 200 libraries in the standard library? 200 modules. Do I know all 200 intimately? I don’t. Well, what if my editor does? What if my linter does? What if my quick fix editor… You can augment my knowledge of libraries. They could even say, “Tim, it looks like you’re trying to build JSON by hand. Why don’t you use the tag library? The JSON library instead. You’d reduce your code by 85%.” Why would I not want that?
I think the opportunity for AI to augment the people doing the work is huge. On the other hand, going away from cancer and to male pattern baldness, by the way, this is a good joke because both of us are bald men.
Shane Hastie: Absolutely.
Tim Ottinger: You can’t tell on audio and we’re happy in it. So go into that, of course there’s people who want to pay to replace and reduce their programming staff, and they’d like to have it done automatically. And this has been forever, right? Remember the CASE tools? You just draw the pictures and it’ll write the code and a million code generators and low code. Remember the 4GLs? We’re back to low code, no code. It’s Microsoft Access and 4GLs again. And you know what? Those are good things. Those environments are good ideas. Man, was anybody more liberated than the people who first got their copy of Lotus 1, 2, 3 or Excel or Quattro? You used to have to wait for somebody to go write code, and now you can do your own spreadsheets and you go to Microsoft Access and a project that you would’ve had to sponsor and fund for a year, you can kind of approximate it with bad code in days. We should do that. This is a good idea.
When we realize that that’s not so grand, wouldn’t it be cool to have tools to help us understand what it does and decipher it and untangle it and build something that you can turn a product into, something you can commercialize and make money with and make a lot of people’s lives easier? I think that kind of AI thing is a good idea. I think augmenting human beings is the proper role of AI. I think replacing human beings, it’s a pipe dream, is probably going to do a lot more damage than it does good. And you know, smart searches are smart. I like smart searches. I’m okay with that. I wish that they could only tell the truth. I wish they didn’t make up truthy sounding lies. But if we know that it’s a liar, then we’d know not to trust it and we can probably handle that okay, too.
Also, remember, there’s a meme going on the internet in Twitter. I saw a post that said, “We thought that the information age was going to come about and computers would do all the hard, tedious work to free us up to make art and music and human advancement. So why are the computers making the art now?” Well, we work hard. Hopefully we’ll know. It’s new. It’s like social media is evolutionarily new. People are just trying to come to terms with what it’s for and how to use it. We’ve only had it for one generation’s lifetime, so yeah, of course, it’s a mess out there. And then there’s brutality and meanness and cruelty and shallowness and eyeball-seeking clickbait nonsense and commercial emotional manipulation. And what? Yeah, of course there is. It’s new. We’ve got to make a mess of everything before we figure it out. Look what we did with industry. But on the other hand, give us a bit of time. Give us a generation or two. Help us assimilate and become comfortable with these new things and we’ll find ways to make them service.
Shane Hastie: It’s an optimistic note.
Tim Ottinger: Yeah, I’m a realistic optimist.
Shane Hastie: Some really interesting points and thoughts through there. If people would like to continue the conversation, where do they find you?
Tim Ottinger: I am super easy to find, so if you’re on LinkedIn, you can find Tim Ottinger on LinkedIn. There’s Tottinge, T-O-T-T-I-N-G-E on… What is it called now? Twitter? X, that thing. I’m on Mastodon as Agilator. Of course, if you go to the Industrial Logic website, industriallogic.com, that’s probably a great way to go. You go to the blog. I’ve written quite a few blogs there. You can go to the people page. It gives links to me on different media. I’m on Bluesky. I’m on Threads. I’m everywhere. I am inevitable.
Shane Hastie: Tim, thanks so much for taking the time to talk to us.
Tim Ottinger: Thank you, Shane.
