| Why Classic Software Engineering
| 
|
|
| Why Classic Software Engineering
|
|
Note: This is a formalized version of a speech delivered by Mike McShaffry, a programmer at Origin Systems, Inc., Austin, Texas, on LegendMud on September 17, 1996. A section of Q&A follows the body of the lecture.
Software engineering can be defined as "...the technological and managerial discipline concerned with systematic production and maintenance of software products that are developed on time and within cost estimates."
This definition was pulled from a popular textbook on software engineering, "Software Engineering Concepts" by Richard Fairley. The above definition mentions nothing about the quality of the software or the working conditions of the development team. Both of these issues should be of paramount interest to the developer, as quality is the life blood of any development house and the development team is your hardest to replace resource.
What I hope to focus on in this short study of software engineering are additions and modifications to these more well known procedures. Software engineering breaks the project into planning, cost estimation (an important subset of the planning process), software design, verification and validation, and finally maintenance stages.
Having a complete development plan is good for developers because there are no questions to ask. You get your daily task in your e-mail and it takes you exactly 6.5 hours to complete (1 hour for lunch and 30 minutes of misc. time) and you turn it in at the end of the day. For some kinds of software this is actually possible. I was involved in the design of a revenue control system for parking lots, and every last screen and input key and algorithm was designed before hand.
This is all fine and well when there are no questions about the development path. In computer games this is rarely the case. I'll talk more about how research affects the cost of development, but realize that it also affects the ability to create a comprehensive plan.
Something that I've never seen discussed in planning games are backup plans. Anyone in the military knows that a battle plan rarely survives contact with the enemy. I think I could make a good case for saying that two things are required to win your battle - a solid initial plan and an intentional willingness by you to be flexible about the plan. In the real world any of the aspects of your plan are subject to violent change without notice. Your ability to respond to those changes (you don't think it's going to happen only once during the project) quickly and correctly will determine the overall success of your project.
The project plan must identify important resources and provide backup plans if those resources are unavailable. These resources should include key personnel, special equipment, or technology dependent on the outcome of research. The more important the resource, the more effort should be applied in the earliest stages of development to secure it for the life of the development cycle.
For goodness sake don't divulge your plan at trade shows too early. The computer gaming industry is filled with young ambitious people who will find ways to work 30 hours a day to beat your plan. Competition moves very quickly so don't make the mistake of underestimating anyone.
Most development plans include milestones. The classic components of a milestone include an expected due date, a complete description and a non-subjective test for completeness. The last one can be tough; imagine a milestone that reads "Combat system is fun to play". Obviously this milestone cannot be tested accurately and can slip indefinitely. It is better to refine it somewhat to something like "All ranged weapon attacks and defenses functional". Even though it doesn't say anything about it being fun (which is the overall goal of any computer game); that aspect must be implied throughout every milestone and can be stated in the project goals.
One more thing about milestones is that they will probably change somewhat throughout the project. The more detailed the milestones, the more likely they will need to be modified as the battle plan is modified. This is especially true if each team member has a specific contribution to a team milestone. If the milestones are planned well and are relatively flexible thay can be a huge motivating factor. If milestones are not thought through and are inflexible, the inevitable slips will result in a loss of morale and productivity.
Most classic cost estimations use experts to make successively refined guesses from information about the breakdown of the entire project and the project team. These guesses are milled and modified with factors that take into account things like "effort multipliers". Differences in opinion are handled by averaging (or even weighted averaging).
(Interesting note: in most game proposals I have witnessed the expected costs were included before a team was assembled. This made the cost estimate a second order guess!)
These methods cannot account for the most unpredictable factor of all.
The largest unpredictable factor in software cost and schedule estimation is the amount of pure research oriented work that must be done to meet the goals of the project. Anyone who tells you they can estimate costs associated with research oriented projects is either naive or also sells beach front property in Arizona. A gross exaggeration will make my point clearer: imagine your boss walking in to your office and requiring you to estimate the costs involved to find a cure for AIDS; by the end of the day if possible. This is why pure research cannot have a final price tag.
Software is generally not pure research. Computer games have many components that require absolutely no new research at all (by the way, don't confuse design with research - design has a clear path that will lead to an unknown end, research has an unclear path to reach a known end). Some of these components include user interfaces systems, file input/output, network communications, simple graphics primitives, random number generation, and so on.
(Another interesting note: The computer game industry is very young and generally employs young people. An interesting component of the overall cost of computer games is that of developing technology that already exists. This cost is never included in the original plan. I have personally seen programmers redevelop something that was readily available because "it helped them understand it." It seems to me that any organization that finds a positive way to discourage this behavior will succeed in bringing overall costs into line.)
Because most computer games do something special to be competitive there is a pure research component to the overall cost, but it is generally impossible to estimate it with any accuracy. An estimate must be made to appease executive management and marketing, as long range plans are vital to the organization as a whole. These mutually exclusive goals challenge any computer game company.
Don't confuse a sequel with a maintenance version (even though Microsoft seems to find ways of making us pay for Windows over and over again). Most computer game companies offer bug fixes on the online services rather than remanufacture the software.
The result of this absence of the maintenance phase of the life cycle vastly changes the way the code is written. The first place programmers can cut corners when they are under pressure is the maintainability of the code they write. It takes more time and energy to come up with a reusable documented solution than a one time special case hack. Most programmers really appreciate their own hacks and other's elegant solutions, meaning that the only code a computer game programmer likes is his own! Seriously, the intense competitive environment present in computer game companies is necessarily tolerant of hacking because maintenance is usually a nonissue.
It seems to me however that a huge opportunity is lost here. Every attempt that I've seen to consolidate common code into libraries that are used by more than one development team has failed. I attribute this failure mostly to a fearful attitude of each team to "let them touch our code." This attitude was probably fostered through the intense environment! You can see the viscous circle this creates: programmer is overworked so he cuts corners so his code is bizarre and unreadable and cannot be used and shared by others, causing them to be overworked too!
I've often thought that my true calling was in the plumbing industry where these things don't happen.
Fundamental design concepts such as abstraction, information hiding, clear structure, and modularity are all critical to large software projects. It is easy to see that while a one person project can virtually forego any of this a five person project can be thrown into chaos without them.
All of these techniques seek to break down a large problem into smaller manageable pieces. When this is done for a software project each piece or module must integrate with its neighbor modules so the whole thing functions as a seamless entity. The inherent problem with this concept is that it assumes that most modules are not time critical pieces of code.
Any piece of time critical code will forego any and all organization in favor of speed. This is true in any industry that writes time critical software. While it is true that the risk of a slow computer game is less than the risk of a slow aviation system, the eventual outcome may be shared by the computer game company and the airplane.
Most programmers agree that non-time critical should code follow the hallowed tenets of software engineering as rigidly as possible. No 'gotos' anywhere, lots of internal documentation, modular functions that do exactly one thing and never divulge how they are doing it, and lots of internal documentation in three different languages. These same programmers are the first to use global variables by the hundreds if it means getting one more frame per second out of the renderer.
In a perfect world the entire system would be developed without breaking a single rule in the software engineering canon. Only after implementation was finished on all modules would the bottlenecks be identified and intense rewriting and optimization 'ruin' the insides of some modules. The external interfaces to this module would probably change slightly affecting only minor parts of the other systems.
As you know we do not live in that perfect world and the only place that you get to program like that is in Redmond, Washington. Everywhere else programmers pretty much know where the bottlenecks are going to exist and write code from the beginning that is somewhat optimized and definitely messy. As the project develops there is some entropic force that increases this messiness until even the original programmer has a hard time with it. Given that, it is definitely best to begin the software implementation with the tightest rules enforced, because it only goes downhill from there!
Verification: "Are we building the product right?"
Validation: "Are we building the right product?"
Take a look at the different methods used in verification and validation of software:
I hear many complaints about buggy software; especially about computer games. One of the reasons this is true is a lack of strict adherence to accepted testing techniques. Rigorous adherence means that there is a paper trail for each test and a manager who verified the test was completed.
A bug in any piece of software can manifest itself as an annoying irregularity or something serious like destruction of data. Depending on the software, a bug may even cost human lives. A computer game bug will always fall into the annoying category no matter what it is, because "it's only a game."
The good news about that is that the development costs for games are lower when compared to other kinds of software. Granted, consistent failure to publish quality software will eventually force a company into bankruptcy. If more time were added to get rid of every last bug in game software, the added development costs would be passed on to the consumer, but more importantly the games would take considerably longer to develop.
Since computer games have some lee way in the "features" that were unintentionally implemented either less time is spent on the project or additional features are added. Certainly software engineering gurus would have a very hard time with working in this kind of environment; it would drive them mad.
Any project plan (software or otherwise) must be flexible to change. The best way to be flexible is to think like Murphy and have backup plans that cover probable worst case scenarios. The more questions that remain unanswered when the project is planned the less reliable the plan will be. Meteorologists deal with this problem by putting a percentage chance on their predictions; and project planners could benefit from this lesson.
Software cost estimation follows from the planning process and is every bit as unreliable. A great factor in determining the reliability of cost estimation is the amount of pure research that must be successfully accomplished to meet the goals of the project. In a perfect world no project that had any amount of pure research would be held to a schedule deadline. As I have said before we do not live in a perfect world and the demands of company finances require us to buck up and make estimates anyway.
Software design is a subject held the most dear by staunch software engineers. Good coding behaviors are critical to working in team environments. The fact that computer games have no maintenance phase in their product life cycle and the life cycle itself is usually very short leads technologists to cut corners during development. While this isn't a perfect situation it can be dealt with by beginning a project adhering closely to standard software engineering practices and loosening the standards as the project matures.
Verification and validation is crucial to any piece of software but there are limits to what computer game companies should be held to in terms of investing time and personnel to this endeavor. As the reliability requirements increase so do development timelines and therefore costs. Since most bugs in computer games are never more than an annoyance they are given a lower priority than a bug in an air traffic control system.
The final word about software engineering techniques is that they are useful tools. Knowing when and how to use those tools can make the difference between failure and success.
Somar thinks MrMike was just demonstrating the 'annoying' bug.. cuz this is just a game..
Somar's hand darts up.
'If we could please have people with questions raise their hands, not just ask, and ask only when called on,' Ptah says.
MrMike points at Somar.
MrMike looks at Somar questioningly.
'I'm thinking of a career in game programming, are there any special classes you'd recommend for this or just the standard computer science curriculum?'
Somar says.
McDougan raises his hand.
'The best thing to do is go heavy on math - I would suggest a math minor,' MrMike says.
MrMike says, 'I would also start to specialize in a particular field in computer science as early as possible and get really good at it,' MrMike says.
Somar says, 'Any specific colleges you'd recommend for their programs?'
'Computer programmering is becoming a field of specialists instead of generalists,' MrMike says.
MrMike says, 'No speiccific colleges...but most of my very best knowledge came from working here at Origin instead of my Alma Mater, Univ. of Houston.'
MrMike points at McDougan.
MrMike looks at McDougan questioningly.
'How would you deal with bugs in a more long term game such as this?' McDougan says.
Kai raises his hand to be called on after McDougan.
'What do you mean?' MrMike says.
Somar has a question after mcdougan and kai are done.
'He wants to know what you'd do in my position ;),' Sadist says to MrMike.
Ptah says, 'I think he means the fact that a mud is always under development to respond to pressure from players and enw ways of using the game, given a much longer play life.'
'Well, I've seen the coder comments, and they've uncovered crashing bugs to reveal hanging bugs, and all sorts of weird things,' McDougan says. 'This mud USED to crash only once every three days until they fixed that bug, now it crashes.'
Rusalka says, 'We're always in updates.'
Shalindra nods her agreement with McDougan.
'Gail fixed something, then it broke, and then she fixed it again, and it worked sorta, then she fixed it agan, and now it's even more broke,' Shalindra says.
Shalindra giggles.
'I could only speculate what the proper prioritization of bug fixes should be on a persistent project,' MrMike says.
'In a nutshell,' Shalindra says.
'Three times a day,' McDougan says.
'But I imagine that given any two bugs one can be arguably more critical to the overall quality of the product, and should there fore be fixed.'
Ptah would add that players would prefer the mud to be up with crashes than not up for extensive testing periods when a bug is uncovered. :)
Point nods his agreement with Ptah.
MrMike says, 'Also, in more traditional software bugs have an annoying tendancy to effect systems of different fragility and therefore have more inherent risk in attempts to fix them,' MrMike says.
Ptah says, 'Which considerably harms the rate of fixing, and increases the chances of incorrect or insufficient fixes.'
McDougan says, 'I wouldn't mind if it hung a while while they worked on that bug.'
Rusalka nods solemnly.
Rusalka says, 'So we try and fix one and something gets out of whack.'
McDougan says, 'Hmmmmmmm....'
'All of these factors should be taken into account when determining which bugs get fixed and which ones remain 'features....',' MrMike says.
'Hangign is worse,' Rusalka says to McDougan.
'Fine, but MrMike doesn't know anything about it, and a lot of players would disagree with you, unfortunately,' Ptah says to McDougan. 'I suggest we try to keep on general topics, not questions that he can't possibly know an answer to.'
Ptah smiles at McDougan.
McDougan says, 'Sorry.'
Red1Guest nods his agreement with Ptah.
Somar applauds Ptah's quick thinking and good judgment.
Ptah promises that we try to fix critical bugs as soon as we can :)
MrMike points at Kai.
Kai says to MrMike, 'Changing to a financial front, would the cost assoociated with taking the extra time to debug games be worthwile when we have to pay more for them or would the extra cost be nonexistant?'
MrMike says, 'The extra cost could be double or even triple the current cost.'
Point wrinkles his nose distastefully.
MrMike says, 'For an added benefit that is debatable.'
McDougan frowns.
'So, $120 for the average game?' Kai says to MrMike. 'Is the cost worth it for three or four fixes we could get off the internet?'
'I think everyone would love a perfect game that has no flaws whatsoever...' MrMike says. 'But it just isn't reasonable to ask a development team to deliver the same kind of reliability that exists in other kinds of software.'
Kai nods his agreement with MrMike.
Kai thanks MrMike heartily.
MrMike says, 'After thinking about it, my estimate of 3x may be woefully hort.'
Kai shudders.
MrMike points at Somar.
Somar says to MrMike, 'What amount of education is usually required before a pogrammer is hirable by a large company?'
'Not to mention most development teams would probably quit if forced to poor over code hunting for obscure bugs for another 6 months or a year after having finished a product,' Sadist says.
McDougan shudders.
Shalindra comforts Sadist.
'It completely depends on the company,' MrMike says. 'IBM won't look at you unless you have.'
'A degree,' MrMike says. 'Origin has hired people out of high school and they have become.'
'Fortunately, there's some [small] companies that hire people based on how good they are, not how many pieces of paper they have,' Shalindra says.
Somar says, 'For example, in Origin mebbe, how many years of college usually must be gone through before being hired?'
'Key members of our staff. ' own companies in some cases),' MrMike says.
Rusalka bets it depends on the person.
'3 of the 4 programmers on our team were in or had just started college when hired, and only one has a degree,' Sadist says.
MrMike nods his agreement with Rusalka.
Sabella raises her hand.
MrMike says, 'Any more questions?'
MrMike points at Sabella.
MrMike looks at Sabella questioningly.
' 'When you were talking about cost estimation (i think that's where it was) you said disputes were settled by averaging or weighted averaging,' Sabella says.
Sabella says, 'Could you explain that?'
MrMike says, 'Sure.'
'When cost estimates are given by different people ,' MrMike says. 'They are rarely equal,' MrMike says. 'The differences are usually resolved by averaging the different numbers together!'
Rusalka nods solemnly.
MrMike says, 'Sounds kinda silly to me really.'
Sabella says, 'O i see.'
Blue1Guest chuckles politely.
MrMike says, 'The weighted average .'
'Means that some peoples estimates carry more respect and have a heavier influence on the final estimate,' MrMike says.
MrMike says, 'Any more?'
'If there were one thing you could change about the normal development cycle for games, what would it be?' Sadist says.
'Give me all the time I want and all the money I want and fun people to work with,' MrMike says.
Sadist says, 'Good answer ;).'
Shalindra giggles.
Rusalka giggles.
Somar's hand flies upward again.
Dekker wishes MrMike good luck!
Sadist could use some more time right about now.
'Ptah and kaige are not fun?' Shalindra says.
Somar giggles at Shalindra.
Sadist says to Shalindra, 'They're on my team, not his :).'
'Oh!' Shalindra says.
MrMike says, 'Seriously, tho...'
MrMike says, 'I would change the heavy emphasis on quarterly performance.'
'This is something I did not mention in my lecture, but some companies, (remaining nameless), require that some projects meet arbitrary quaterly deadlines when. It might mean all the difference in the overall quality of the product.'
'Many people were upset with the initial release of Ultima 8, ,' MrMike says.
Madcat nods his agreement with MrMike.
Blue1Guest chuckles politely.
'And the main reason this was the case is Origin's desire to meet our first EA quarterly (actually end of year) deadline,' MrMike says.
Rusalka nods solemnly.
MrMike says, 'All it would have taken to fix 80% of the most complained about problems was about 2 weeks of work,' MrMike says.
'So things got a bit rushed,' Rusalka says.
'Yipes!' McDougan says.
MrMike points at Blue1Guest.
Ptah says to Blue1Guest, 'If you could, introduce yourself as well, when you
ask your question :).'
'Well first i guess i should introduce myself since i'm just a guest and don't have a name,' Blue1Guest says.
Blue1Guest chuckles politely.
MrMike looks at Blue1Guest questioningly.
'I commonly play as Dranor in most locations, but anyway on to the question,'
Blue1Guest says.
'I've been interested in becoming a computer programmer for quite a while and have been involved in some coding projects such as another mud, i was wondering,' Blue1Guest says.
Blue1Guest says, 'A) if this is something that could actually help me get a job in the field and b) if you have any suggestions for me.'
MrMike points at Ptah.
MrMike points at Sadist.
Ptah chuckles politely.
Somar applauds MrMike's quick thinking and good judgment.
MrMike says, 'It certainly helped them!'
'And Rufus, and Kaige,' Sadist says.
Blue1Guest chuckles politely.
Rusalka points at the absent ruf
Sadist chuckles politely.
Rusalka nods her agreement with Sadist.
MrMike says, 'As far as other sugegstions.'
'The answer to that part, since you can't be expected to know, is that I work at origin, and so does Kaige, another imp here, as well as other Legend staff, largely thanks to work on muds,' Ptah says to Blue1Guest.
'Ptah and sadist (and kaige and rufus) started legend, then went to work for Origin,' you say to Blue1Guest.
MrMike says, 'I have seen many at Origin get thier start.'
MrMike says, 'Not by submitting resumes, but by submitting something of thier work.'
'I have seen some really great programmers here who got thier job by reverse engineering,' MrMike says.
Sadist says, 'Actually ALL of our programming team had some involvement with muds/mushes/etc.'
MrMike says, 'A large portion of older Ultima games.'
'Reverse engineering?' Rusalka says to MrMike.
MrMike points at Somar.
Sabella raises her hand.
'Taking the finished product and decompiling and understanding the code,' Madcat says.
MrMike says, 'Reverse engineering - looking at something and figuring out how it works and copying it....'
Somar says to MrMike, 'How would one go about applying for an entry level programming position usually?'
Rusalka nods solemnly.
Shalindra wonders how this got so off-topic.
Somar is in senior year of high school and searching for all sorts of possibilities.. =)
Rusalka thanks MrMike heartily.
'One of the best things to do is put together a demo disk of things you've written,' Sadist says to Somar.
'The usual way almost never works.....but you submit a resume and call the human resources department alot and hope,' MrMike says.
McDougan knows these things ALWAYS get off topic- Where have you been, Shal?
MrMike says, 'The best way is to find a programmer that already has a job, and submit somehthing cool that you have done,' MrMike says.
Somar tugs on Ptah's sleeve imploringly.
'Well, it's never gotten off-off topic!' Shalindra says.
MrMike points at Sabella.
Sabella says, 'Our development problems in an on-going game (the mud) are complicated by player issues. But i was wondering at what point your would say putting off release to add new features becomes... counter-productive, and if you would release features that you know will change (given the opportunity to change) anyway to get them out :).'
Sabella hopes that made sense :)
MrMike says, 'Interesting question....'
MrMike says, 'I'm afraid I could only take a guess at this... since my experience lies in a very different kind of software, but knowing what your customer wants is critical to your success,' MrMike says.
'At all stages the best possible situation is one where you can satisfy each and every whim of each and every customer,' MrMike says.
'Since this is not possible, you must prioritize them in terms of benefit, 'Amount of work, and even feasability of the technology,' MrMike says.
'In other words, use your best common sense, make a guess, and go,' MrMike says.
Sabella nods agreement with him.
Coman says, 'Except that working a mud is something like working on a dyamic system with an established user base.'
Coman says, 'You have to balance what everyone wants against propagation of change through the system.'
'True,' MrMike says.
MrMike says, 'But I can argue that I make the same decisions as the software I'm working on goes through our testing department.'
Sabella asked him to ignore that, knowing that he doesn't deal with that situation...
MrMike says, 'I worry about implementing each and every feature they (the testers) desire.'
MrMike says, 'What will it do to the rest of the system...something I cannot always predict.'
McDougan says, 'Hmmmmmmmmmm.'
'My apologies for being late,' Coman says. 'I missed too much of your early explanation. but, our software is also extremely complex, with a user that is constantly changing ...'
'Though in many ways it's easeir for us to change,' Rusalka says.
'There will be log of the lecture on the web page, I'm told,' MrMike says.
Rusalka says, 'All the software is on the system.'
'Sorry Ptah,' Coman says.
Rusalka nods her agreement with MrMike.
'Really?' McDougan says. 'Hey! I'm on the web page!'
'Any more questions....we have a few moments more...' MrMike says.
Ptah notes again that MrMike is not really familiar with the mud world and hence process differences aren't really something he can address well.
Siachet giggles at Somar.
Red1Guest nods his agreement with Ptah.
Ptah also notes that we are still trying to avoid spam and emotes and so on. MrMike points at Coman.
MrMike looks at Coman questioningly.
'Lol. to finish that first question :). at some point, you have a single, integrated design. working, tested. you freeze it at that point. then, iterate through the design process with a "build" established.' Coman says.
'How is your methodology counter to that technique since it's a pretty well established method of process?' Coman says.
'Well if there are no more questions.....' MrMike says.
MrMike says, 'It isn't exactly counter to that technique.'
'It is however inherently more flexible than,' MrMike says.
'Many software engineering techniques as they are taught,' MrMike says. 'Think of it like software engineering for the real world!' MrMike says.
'I have thought about it,' Coman says. 'I have designed extremely complex software systems for the real world, and in situations designed to reduce costs to a bare minimum.'
MrMike says, 'Go on..."'
'The main failure of game s/w development companies that i've found is that they want to promote the "heroic" development and forgo the process,' Coman says.
'I disaggree completely,' MrMike says.
MrMike says, 'There is a method to our madness....(sometimes I wonder).'
'And our industry is populated with energetic inexperienced people,' MrMike says.
MrMike says, 'The entire industry is less than 10 years old!'
'So there is a kernel of truth to what you say. But I think it has more to do with ignorance rather than stupidity,' MrMike says.
Coman says, 'That is one of the major problems with software engineering (the process - not the programming), there is too little of an established database.'
'SSI though for example, brought in experienced wargamers to help with design,
'Well I have enjoyed this immensely and I hope to do this again,' MrMike says.
MrMike says, 'I'm sorry to run off but I have a hungry dog at home who thinks I will never ever return!'
'Thanks you all once again,' MrMike says.
'Many of the aspects of your questions were sort of addressed in the talk proper, actually,' Ptah says to Coman.
'He talked about how some of the aspects of traditional software engineering product cycles simply didn't exist, and others exist not accounted for in the traditional model when dealing with games,' Ptah says to Coman.
MrMike says, 'Goodbye all.... .'
To send e-mail to the author, click here.