Are there many errors in AutoCAD?

Quality assurance is achieved through multi-stage AutoCAD testing, which does not begin when programmers have written a code in the form of pieces of a program or a function. Rather it starts when product managers and marketing experts have decided what we have to do. We do quality testing at each AutoCAD development stage. As soon as marketing department has determined the required functionality, we check the MRD (Marketing Requirement Document), which they prepared as majority of specialists in our department have a good knowledge of AutoCAD and CAD industry in general. They know, for which purpose AutoCAD should be used, whether it be architecture, design, electrical engineering, landscaping, etc. Upon reviewing MRD our experts develop a vision as to which extent a particular functionality is necessary and can be implemented, and suggest possible changes or other methods to resolve any issues. It takes place at the concept level and it starts from here. At the next stage, when marketing finished their part and designers begin discussing how a particular function should work (command prompt, dialog, ribbon, etc.), we test their specifications and confirm whether they are correct or not, whether something must be changed, assess how much users may be “interested” in working in a particular manner. Based on our industry experience, we try to influence how the end product should look like.

Then our main task begins – to verify directly the quality of a program, a code. It can be done using several techniques. First, there are the so-called manual or black box testers – people, who know the program and simply try all freshly implemented functions to see how efficient they are, how they interact with already existing functions in the program, etc. Such testing also includes file saving and version control support.

Second, there are the so-called white box or API testers – programmers who work at the QA Department. They check functionalities that will be needed by third-party developers to create all sorts of AutoCAD plug-ins. They also work with the code and report to manual testers if a particular part of the code can affect some other functionality that must be checked.

These are the two main areas of AutoCAD testing. There is also a rather significant, one can say, a by-side part of testing – automated testing. We have automated up to 100,000 AutoCAD test cases, and such kind of testing takes place every week. There are a lot more manual tests, their number depends on the function that we are testing and how it interacts with other functions. So far, these are the three main types of testing in terms of functionalities.

After our experts have written the documentation, we also have to test it to verify to what extent it is correct.

Regarding members of staff in our Department, I’d rather not mention the exact figure. Let me just state that it comprises dozens of experts.

In our Department we test AutoCAD platform, as well as AutoCAD LT, DWG TrueView, AutoCAD for Mac – this is in terms of the platform. The department also does the testing of AutoCAD Architecture, AutoCAD MEP and AutoCAD Mechanical.

Answering your question, I’d like to talk a bit about how AutoCAD is developed so it can be localized – because before localizing one must be certain that it is possible. To this purpose, we set special requirements for the program and our programmers know what they must do, and we check this. What we test is called globalization. Unlike localization, globalization is testing a program to ensure that it can be successfully translated in other languages. For instance, with English AutoCAD we always test several already translated versions: German, Japanese, or Chinese ones. We also check other language versions on a regular basis. I’ll be honest with you: the experts of our Department cannot know all these languages and we can’t always check everything with 100% accuracy. If I remember correctly, there are no Italians in the Department, but we still test Italian version to be certain that it will work. I’d like to point out that verifying translations is not part of our duties – there is a localization team that takes care of this.

I am not sure what the exact ratio is. It is extremely difficult to give you the percentage. Not 100% and not 0%, but something in between :). We devote much attention to beta testing because regardless of a highly professional level of our team of engineers we will never be able to simulate the whole lot of variants for installing our software on thousands computers with different configurations of companion programs, tuning, etc.

Honestly, we are not always able to reproduce error reports, which we receive from our users, at the internal level. This is understandable - users are not professional testers. Nevertheless, we analyze absolutely every problem and look for those users that, firstly, use AutoCAD in real-life design projects, and, secondly, with special configurations that we were unable to test.

This is a very important part of AutoCAD testing. Moreover, there are different levels of beta program: an ordinary beta, where people download software and some use it once, while some use it very actively and interact with us on the beta forum. The next beta level is when we recommend beta-testing companies to use a pre-release version in particular projects. With these people we work one-on-one, and help them if any problems arise. Sometimes we release hot fixes and take their comments very seriously. On the other hand, it is easier for them to explain the core of the problem to us and determine the right priorities.

OOne person, with who I work, is based in Moscow and he is one of our best beta-testers, under the “Gold beta tester” program. He has provided many comments which have been very helpful in improving the quality of our software.

Most of the testing programs that we use are standard. They were designed specifically for software testing but we put them in our own framework, that we created ourselves to make testing fully automated and much more efficient. As for the system, we use internal systems and internal processes that have been developed and adjusted throughout many years. Overall, our company uses agile system development for AutoCAD, which in some cases is used to the full and in some cases – only partly. And we are a part of this general system.

It is an interesting question as the presently fashionable agile system development implies that people are based in the same office or in the same big hall without any compartments and work together. There is no secret that AutoCAD is developed across the globe and we put every effort to bring testers and programmers together. Often we succeed, sometimes this cannot be done (for instance, when a testing engineer is located in the USA and a programmer is based in Singapore). In this case we must search for such approaches that these experts can work jointly and efficiently in spite of enormous distances. To interact from different locations, we use Microsoft Communicator, which is very similar to Skype, and we use it constantly. Also we are always in communication via e-mail. Back to the beginning – we work with programmers very closely; sometimes literally sitting shoulder to shoulder with them, but it is not always possible.

We interact under the testing program. There is a special program for third-party developers, that allows updating the existing programs to the latest beta-versions under the framework of Autodesk Developer Network and sends error messages and suggestions regarding software interface development to Autodesk developers. In this case ADN is the link between us and developers. If necessary, we would work directly (it does not happen often but it does take place, especially with “gold beta testers”).

We have a special feedback page on our web-site, where it is better (but not necessary) to write in English. Upon receiving a message about an error, its description is sent by e-mail to a group of people who will study the problem. (It stands to mention that we not only receive many suggestions but also messages with … abusive, offensive language, which, to tell the truth, we are not willing to respond to at all. On the other hand, when trying to answer rude messages politely, in 99% we get responses like “sorry, I did not think that anybody will read this”.) After completing the form, a user receives an automated message, expressing our thanks and reassuring that the message will be read. We are unable to respond to every message but we do read 100% of such messages. In case of specific problems we often respond and sometime enquire additional information from our users.

f a program crashes, users also can send an automated error report that will be processed by a special program on server database. All such reports are grouped according to the error field and then we look at these groups, identify priorities and start working with the largest groups. Therefore I would like to ask users: please, send such reports to us as we identify priorities depending on the quantity of errors. In some cases we experience difficulties reproducing the error and if a user gave us the e-mail address, we can send clarifying questions. There is another reason why you should give your e-mail address in such kind of reports – those who did it will automatically receive information about updating the program that corrects the problem in question.

I always ask users that if they can they should indicate specific steps that led to the error, and I would like to take this opportunity now to call upon the readers of isicad.ru!

We consider all variants and make a decision depending on a particular error and how many errors we have to deal with. There is always a threefold framework: timing – quality – functionality; and a decision is made based on a combination of them. In the past year there were at least two cases when we had to postpone software release as it was not ready. On the other hand, there are cases, when functionalities were cut off, if based on testing we found that it was not ready to be released and in such situations it is better to cut off a particular functionality and release the software without it rather than wait for improvement.

I very much wish I can say “yes”, because in this case my life would be easier. The number of staff is increasing every year. There is no decisive answer whether it is related to increasing number of users or the growing quantity of tested products. Everybody understands that the size of company workforce, including Autodesk, depends on the level of sales and budget. Plus the environment: during economic crisis – the company is going through a more difficult period, during recovery – life is easier, and it follows a sine-shaped trajectory. Since I started working for Autodesk (14 years ago) the number of staff has increased.

AutoCAD has a smooth and efficient work system, when we know roughly what we can do, what level of quality we can reach and how fast it will be done - we have many so-called check points. And if in any particular check point we know that we have achieved what we planned, we can state that ultimately everything will be fine. The whole process is adjusted to meet such check points.

As for AutoCAD, some releases are better, some tend to be worse, but the whole process is sufficiently smooth. And releasing new software is always an untrod path in spite of colossal experience accumulated in other projects.

The most interesting and unorthodox was releasing the first AutoCAD version for Mac (2011). These were absolutely new technologies, new approaches, and new automation tools. Everything was new and it was very interesting. On the other hand, however, it frustrates smooth processes, which makes you quite nervous.

We have a special team dealing with this. We pay considerable attention to the issue (recovery of damaged files) and in case of a problem we always try to recover data. With every our release, data recovery functionality is getting better. In every case when the program crashes, we devote significant attention to data integrity. In terms of other programs, testing API we endeavour to ensure the highest possible data security, but it is not always possible. If we do not work with these companies through ADN, we cannot test these files as we do not have the tools to test a particular functionality. It is especially true for the software for DWG-files that is not based on Autodesk in-house components.

In our team most testers are former users. When we hire a team of manual testers we chose people who know how to work with AutoCAD rather than those who are able to test software. We always face a choice: a software tester or a professional software user. We would always prefer a person who knows AutoCAD and the industry in general. That is why I was hired by Autodesk. It is my AZLK experience, where I first learned the basics of AutoCAD, and then improved my knowledge to a more professional level, that helped me find an AutoCAD-related job relatively quickly when I came to America. I subscribed to AutoCAD User Group - a “professional hobby group” that had no connection to Autodesk. Thanks to the skills acquired at AZLK I found my first job in the US and later I was employed by Autodesk. The foundation was laid at AZLK.

I am not just wondering about it - I am fully convinced that serious changes will take place. Five years ago we could not imagine that we would be able to do such things on smartphones. Ten years ago the Internet was already at full swing but I don’t think at that time there was considerable network cooperation. Now, in 2011 everything is in clouds. I am certain that very soon there will be fully-functional designer programs for smartphones. If in ten years AutoCAD is designed for Internet or mobile devices rather than for desk computers – it will be a different product. Seeing how the world around us is constantly changing, we observe these processes, assess them, and experts put lots of efforts in this field studying user suggestions and global trends. In my opinion, engineering design is more conservative in comparison with Internet-developments or mobile development. Drawings made two hundred years ago are not much different from the modern ones. Certainly, there are people who still use drawing boards but it does not mean that the future is there. The same works in our field: if it becomes necessary to stop releasing AutoCAD for PC or modify it to work in Internet or write something completely new – all this will be considered and is already being considered.