AIware Leadership Bootcamp 2024
About the AIware Leadership Bootcamp
Zero to Hero in under a week! The AIware Leadership Bootcamp is a unique opportunity to learn from and interact with leading industrial practitioners, researchers, and academics. The bootcamp is designed to provide a holistic understanding of AIware development—going beyond simply teaching how to build software with Foundation Models, e.g., Large Language Models (LLMs), to exploring the latest research, innovation, and the broader AIware ecosystem. It’s also about empowering leaders who will drive the future of AI-powered software.
“Software for all and by all” is the future of humanity. AIware, i.e., AI-powered software, has the potential to democratize software creation. We must reimagine software and software engineering (SE), enabling individuals of all backgrounds to participate in its creation with higher reliability and quality. Over the past decade, software has evolved from human-driven Codeware to the first generation of AIware, known as Neuralware, developed by AI experts. Foundation Models (FMs, including Large Language Models or LLMs), like GPT, ushered in software's next generation, Promptware, led by domain and prompt experts. However, this merely scratches the surface of the future of software. We are already witnessing the emergence of the next generation of software, Agentware, in which humans and intelligent agents jointly lead the creation of software. With the advent of brain-like World Models and brain-computer interfaces, we anticipate the arrival of Mindware, representing another generation of software. Agentware and Mindware promise greater autonomy and widespread accessibility, with non-expert individuals, known as Software Makers, offering oversight to autonomous agents.
This bootcamp focuses on educating participants about the full spectrum of AIware development and innovation—from the current state of Neuralware and Promptware to the emerging paradigms of Agentware and Mindware, where intelligent agents and human oversight come together. As future leaders, participants will gain the skills and insights necessary to navigate and influence this evolving landscape.
To register for the event, please follow this link. The event will be held from November 3 to 8 in Downtown Toronto Canada at the Queen's University Downtown Toronto campus, a state-of-the-art facility with stunning views of the Toronto Harbour. Spaces are limited, so early registration is encouraged to secure your spot.
Why Attend?
Unlike other bootcamps, our bootcamp emphasizes a holistic perspective on AIware, integrating bleeding-edge research and next step directions with practical applications. With panels and presentations from leading experts, you’ll explore open research questions, gain insights into the most recent advancements, and understand the evolving needs of the AIware ecosystem. It’s an essential experience for anyone aspiring to lead in the AI-powered software arena.
This bootcamp is not just a learning experience—it’s a chance to create lifelong collaborations and networks that can lead to significant future opportunities. By engaging with the brightest minds in the field, you’ll be at the forefront of shaping the future of AI-powered software.
The bootcamp will also emphasize a hands-on perspective, ensuring that attendees have a more pragmatic perspective in this important and emerging domain of computing.
Curriculum Committee
The curriculum integrates feedback and input from industry and academic leaders from the following organizations.
Bram Adams, Queen's University, Canada Eddie Aftandilian, GitHub, USA Boyuan Chen, Centre for Software Excellence, Huawei, Canada Satish Chandra, Google, USA Filipe R. Cogo, Centre for Software Excellence, Huawei, Canada Prem Devanbu, University of California Davis, USA Danny Dig, University of Colorado Boulder and JetBrains Research, USA Ke Ding, Intel, USA Keheliya Gallaba, Centre for Software Excellence, Huawei, Canada Ahmed E. Hassan, Queen's University, Canada Zhenming (Jack) Jiang, York University, Canada Yoshimura Kentaro, Hitachi, Japan Foutse Khomh, Polytechnique Montréal, Canada Jiahuei (Justina) Lin, Centre for Software Excellence, Huawei, Canada Dayi Lin, Centre for Software Excellence, Huawei, Canada Yiling Lou, Fudan University, China Qinghua Lu, CSIRO, Australia Orlando Marquez, Servicenow, USA Gustavo Oliva, Centre for Software Excellence, Huawei, Canada Ipek Ozkaya, Software Engineering Institute, Carnegie Mellon University, USA Gopi Krishnan Rajbahadur, Centre for Software Excellence, Huawei, Canada Baishakhi Ray, Columbia University, AWS, USA Ben Rombaut, Centre for Software Excellence, Huawei, Canada Tom Zimmermann, Microsoft, USA Haoxiang Zhang, Centre for Software Excellence, Huawei, Canada Lingming Zhang, University of Illinois Urbana-Champaign, USA
Schedule
View the detailed agenda for each day by clicking on the respective buttons below.
FMware 101 - Intro into the world of FMware, SE4FMware and FMware4SE
Registration and morning coffee break
Opening of the Bootcamp
Instructor team
Opening talk and logistics
Challenges and opportunities associated with AI-Augmented software engineering
Zhen Ming (Jack) Jiang
Introduction to challenges and opportunities in the world of AIware focusing primarily on Promptware and Agentware
Break
Building high-quality and trustworthy foundation model applications
Dayi Lin
This session will cover the basics of building a trustworthy FMware application. Topics covered include
- Overview of AIware
- Introduction to the trustworthiness of software and AI
- How do you select the right model?
- How to benchmark a model?
- How to write a good prompt?
- How to prevent hallucination?
- How to debug prompts?
- How to prevent getting or causing harm? - Guardrails
- How to ensure compliance in dataflow?
- How to conduct quality evaluation?
- How to interact with the users?
- How to operationalize the application?
Rethinking software engineering in the AIware Era - A curated catalog of software engineering challenges for FMware
Ahmed E. Hassan
The talk walks the audience through the key Software Engineering challenges in the AIware era and outlines important research directions that could be taken to address them. Topics include
- Overview of Software Engineering challenges in AIware Era
- Challenge 1: Managing alignment data
- Challenge 2: Crafting effective prompts
- Challenge 3: Multi-generational software
- Challenge 4: Degree of controllability
- Challenge 5: Compliance and regulations
- Challenge 6: Limited collaboration support
- Challenge 7: Operation and semantic observability
- Challenge 8: Performance engineering
- Challenge 9: Testing under non-determinism
- Challenge 10: Siloed tooling and lack of processes
Building a thriving research and developer community within the AIware domain, discussing the essential steps we must take collectively to nurture and sustain a vibrant, engaged community and a brief overview of global initiatives and standardization efforts including:
- CREATE SE4AI
- SEMLA
- BigCode
- AI Alliance
- OPEA
- AIware
- LLM4code
- MSR IEEE Agent Standard
- AIBOM
- AIwareBOM
- LF Model Openness Framework
Lunch and lightning talks
The data flywheel for FMware
Gopi Krishnan Rajbahadur
This session will go over the different types of data used throughout the lifecycle of an FMware. Topics covered include
- Data flywheel - An overview of the different types of data that is used for different types of alignment including:
- Pre-training data
- Fine-tuning data
- Preference data
- User-feedback data
- Crowd-sourced data
- Synthetic data
- Synthetic data generation techniques and data alignment (Microsoft Phi models)
- QA for Knowledge distillation - Includes data quality and knowledge quality enhancing processes like Deduplication, Information metrics, Removing data and knowledge clones
- Contextualized and goal-oriented documentation to enhance the knowledge quality
- Operationalizing user-feedback data - Leveraging the thumbs up/down the copy action Google DIDACT
- BigCode project - A successful application of the data flywheel
- Crowdsourced knowledge curation - Instructlab
- Importance of multi-modal training data - how code is used including techniques on how different types of data can be generated with domain-specific rules in different scenarios
Break
Rethinking intelligent SE in the FMware Era: Overview of current initiatives
Ahmed E. Hassan and Gustavo Oliva
This talk will discuss current GenAI efforts within the SE context, and future directions on how to best leverage FMs to improve SE activities and maximize ROI on GenAI in a Software Engineering Context
Rethinking software engineering in the FMware era: From task-driven AI copilots to goal-driven AI pair programmers and the curcial importane of humans-in-the-loop
Ahmed E. Hassan and Gustavo Oliva
This talk will discuss how the current task-driven nature of copilots falls short in addressing the broader goals and complexities inherent in Software engineering (SE). Covered topics include
- A vision for an AIware pair programmer
- Theoretical underpinnings for an AIware pair programmer
- Challenges in the path of realizing an AIware pair programmer
- Challenge 1: Speeding up human-AI alignment
- Challenge 2: From prompts to natural inquires
- Challenge 3: Cheaper and smarter code models
- Challenge 4: Leveraging mentoring potential
Reception and lightning talks
Building blocks in AIware development
Registration and morning coffee break
Prompt engineering
Filipe Roseiro Cogo
This session will offer a comprehensive overview of prompt engineering techniques and best practices to build a successful FMware. Covered topics include
- Basics of prompting - How to talk to a FM
- Prompting patterns
- Prompt components
- Prompt structuring
- Prompt decoding strategies
- Fragility of prompts - Manual prompt-tuning lifecycle, prompt formatting and context window sensitivity, few-shot ordering and golden labels
- Prompt anti-patterns (e.g. god prompts) - How to decompose a prompt effectively for success
- Prompt output structuring and prompt debugging
- Compiling prompts for success - prompt optimization, prompt tuning, FM-based prompt mutation and evolution, DSLs for prompt optimization, Intent-based prompt calibration
- Common prompting pitfalls
- Leveraging prompt ecosystems - Introduction to prompt stores and Reddit discussions
Break
RAG engineering
Keheliya Gallaba
This session will cover the steps involved in building a robust RAG (Retrieval Augmented Generation) pipeline. Topics covered include:
- What is RAG? - An overview of Indexing, retrieval, and generation
- An overview of query translation approaches - Multi-query RAG-fusion, Least-to-most, step-back prompting, HyDE
- Query routing strategies - Logical routing and semantic routing
- Query construction strategies
- Advanced indexing techniques for effective RAG - Multi-representation indexing, RAPTOR, ColBERT
- Graph RAGs
- RAG Re-rankers
- Large context challenges: Needle in a haystack, counting stars, data movement and caching/pinning of context in the cloud
- Context vs built-in prioritization
- A reference architecture for enterprise-grade production-ready RAGware
Lunch and lightning talks
Alignment engineering
Gopi Krishnan Rajbahadur
This session will cover different ways of aligning a FM. Covered topics include
- A general overview of FM pre-training
- Fine-tuning - Types of used data: supervised fine-tuning, PEFT, Prompt tuning, soft prompts, Adapter tuning, AdapterHub, Selective finetuning (e.g. BiFit), Reparametrization-based fine tuning (LoRA)
- Preference tuning - RLHF, DPO, RLSF
- Constitutional AI
- Overview of curriculum learning
- Advanced curriculum generation techniques and benefits, e.g. how curriculum learning is used in the Microsoft Phi and IBM Granite models
Break
How is JetBrains leveraging FMware to improve the software development experience?
Danny Dig
This talk will present compelling services and applications featuring JetBrains AI Assistant that power software engineering tasks in the IDE to improve developer productivity.
FMware for 10X developer productivity - Myth or fact
Danny Dig, Meiyappan Nagappan, Bram Adams, Daniel German and Foutse Kohm. Moderator: TBD
This panel will focus on questions about role of FMware in enhancing developer productivity - Ask the experts about separating myth from facts.
Agentware and Responsible AIware Development Day
Registration and morning coffee break
Research experiences and best practices building AIware at JetBrains
Danny Dig
This talk presents cutting edge research on combining the creativity of LLM-based recommendations with the safety and reliability of static and dynamic analysis in the JetBrains IDEs to provide an order of magnitude improvements over previous state of the art solutions for refactoring.
Break
Agentic architectures and workflows
Keheliya Gallaba
This session will discuss what an agent is and the different cognitive architectures that an Agent could use to execute the tasks. Covered topics include
- What is an Agent (e.g. perception, Brain, Action, Environment)?
- Different types of agentic memory
- How does an Agent use tools (Toolformer, TALM)
- How to enable an Agent to plan and reason - Introduction to theory of mind
- Cognitive architectures - Multi-layered nature (recursion, multi vs single agent architectures)
- Agent patterns - Chains, routers, graphs
- Control mechanisms - Static vs dynamic
- Patterns of multi-agent collaboration
- Self-reflection and multi-agent hacks
Lunch and lightning Talks
Agentic development platforms
Gustavo Oliva
This session will overview the various Agentic platforms that one can use to build AIware. Covered topics include
- Multi-agent frameworks/ platforms (Introduction, Comparison, their abstractions). Examples include Autogen, CrewAI, LangGraph
- Agent memory representations and abstractions
- Long-term agents i.e. Agents evolving over time, End-user feedback used for personalization
- Standardization efforts e.g. Agent protocol
Break
Data governance in the AIware era
Daniel German
This talk will discuss about the different challenges data governance and dataset license compliance pose in the AIware era
FMwareBOM - Towards enabling transparency, traceability and compliance with next generation FMware BOMs
Gopi Krishnan Rajbahadur
This talk will highlight the current state of AIBOM and ongoing efforts on FMwareBOM and the potential it holds for enabling compliance, transparency and traceability in the FMware era
Responsible AIware development practices
Qinghua Lu, Daniel German, Gopi Krishnan Rajbahadur, Ahmed E. Hassan and Zhen Ming (Jack) Jiang. Moderator: TBD
This panel will discuss governance structures and legistilations around AIware development practices and potential pitfalls
Industry Day 1 – From cool demos to production-ready AIware - Part 1
Registration and morning coffee break
FMware - Hands on
Instructor team
We will be building a few AIware apps which we will make more complex as we progress through the bootcamp. Every hands-on will progressively build on the concepts of that day to make the day more comprehensive. This hands-on session will guide the participants who will be encouraged to add more features to the app and submit it to the AIware conference challenge. The hand-on session will methodically walk the attendees through
- Setting up a FM locally
- Accessing a cloud FM
- Setting up a basic Vector store
- Building a basic AIware app
- Picking the right FM for their FMware
- Preventing data contamination
- Evaluating your FMware
Break
FMware - Hands on - continued
Instructor team
This session will be a continuation of the previous session
- Prompting strategies
- Prompt anti-patterns
- Debugging a prompt
- Prompt compilers - DsPY
- Fine-tuning an FM with LoRa - Data preparation (including quality)
- Preference tuning
- RAG 101
- Best RAG practices
- Progressively building an AIware app with basic cognitive architecture
- A multi-agent collaborative architecture
- Simple memory implementation
FMArts platform - An FMware lifecycle engineering platform
Dayi Lin
This session will showcase the FMArts platform, the world's only lifecycle engineering platform for creating FMware. We will discuss how software engineering principles can be leveraged in the FMware context and highlight the challenges involved. The session will feature a demo application built with FMArts.
Lunch and lightning Talks
Challenges in productionizing AIware
Ahmed E. Hassan
This session will cover the different types of challenges that industry typically encounters in the different stages of AIware development lifecycle that makes productionizing AIware challenging. Covered topics include
- An overview of the challenges across the AIware development lifecycle
- Testing challenges
- Observability related challenges
- Controlled-execution related challenges
- Resource-aware QA challenges
- Feedback integration challenges
- Built-in quality assurance challenges
- Other overarching challenges
Break
Evaluating AIware
Justina Lin
This session will detail the challenges and techniques that can be used to evaluate an AIware. Topics covered include
- Overview of evaluating an AIware - Importance
- Evaluation primitives - Evaluation with ad hoc vibe checks, benchmarks, manually curated datasets, trace data, data splits and repetitions
- Evaluation metrics overview
- Evaluating individual components of AIware - Agents, RAG etc.,
- Testing AIware - unit tests, summary evaluations, response evaluations, regression testing, backtesting
- AI as judge - Overview, benefits and costs
Excursion Plus Take-Home Task
We will go on an excursion to a place near Toronto (location: TBD) and encourage participants to work on a take-home practice exercise that will be discussed on Day 4.
Day 6: Industry Day 2 - From cool demos to production-ready AIware - Part 2 and AIware showcases
Registration and morning coffee break
Performance engineering for AIware
Boyuan Chen and Haoxiang Zhang
This session will outline the various performance engineering challenges as one optimizes the performance of an AIware for production. Topics covered include
- Transformer decoding 101
- Single model serving challenges and the state-of-the-art (e.g., vllm)
- End-to-end performance engineering challenges in the AIware development lifecycle
- Latency considerations for different cognitive architectures
- Multi-model pipeline serving challenges and vision
- Curated SPE challenges for AIware
- Semantic caching and FM routers
AIware observability
Ben Rombaut
The session will discuss AIware Observability challenges. The topics covered include
- AIware Ops overview
- AIware observability overview - What is it, why is it important and how is it different from classical observability
- The importance of semantics vs raw observability
- Existing observability tools and platforms – strengths and weaknesses
- Evaluation and guardrails - Validating FM responses, comparison of models on golden sets, measuring response time, tracking and altering of tracked observability metrics
Lunch
Not provided by the bootcamp - Participants are encouraged to explore the diverse food scene in downtown Toronto
Leading the AIware Revolution - Ask Me Anything!
Ahmed E. Hassan, Mike Godfrey, Quingu Lu, Danny Dig, Yiling Lou and Foutse Khomh. Moderator: TBD
This panel will explore how effective leadership is shaping the AIware era, with a focus on guiding innovation, research, and strategic decisions in a rapidly evolving landscape. We'll touch on the role of leaders in bridging industry and academia, balancing the pace of applied AI with foundational research, and navigating the development of open models. Whether you’re leading a team or aspiring to, this is your chance to ask experienced leaders about the challenges and opportunities in driving the future of AI-powered software.
Break
Showcases of AIware for SE innovations and demos of developed AIware by camp attendees
This session will showcase works from leading experts in the field, and selected AIware projects that have been developed will be presented.
Closing remarks and future initiatives
Ahmed E. Hassan
Closing remarks and discussion on future initiatives.
Event Venue
Queen's University Downtown Toronto campus, located in the heart of the city’s financial district, offers a first-of-its-kind facility atop Simcoe Place. The lecture hall where the bootcamp will be held is designed with state-of-the-art AV and IT systems to support collaborative learning, integrating more design and infrastructure than a traditional classroom. Beyond the custom-designed lectern, the hall opens up to reveal a picturesque view of Toronto Harbour and the railways, creating an inspiring and dynamic environment for our bootcamp.
Queen's University Downtown Toronto Campus
30th floor, 200 Front St W, Toronto, ON M5V 2X3
30th floor, 200 Front St W, Toronto, ON M5V 2X3
Sponsors
We sincerely thank our sponsors for their invaluable support, which helps us not only enhance the quality of our Bootcamp but provide bursaries for students from underrepresented regions and developing economies.
If your company registers two participants through industrial registration, you can also become a sponsor, directly contributing to expanding access for deserving students.
For sponsorship inquiries, please get in touch with us.
Registration
We are thrilled to invite you to express your intent to participate in our upcoming AIware bootcamp at the Queen's University Downtown Toronto campus. Given the high demand and limited availability, we encourage you to submit your intent to attend by September 1, 2024. We will carefully review all submissions to ensure a diverse mix of participants across geography, industry, academia, and experience levels.
Please note that the final deadline to secure your spot is September 30, 2024.
Registration Fees
The event costs $700 CAD (approx. $500 USD) for students and $1,200 CAD (approx. $800 USD) for professors and researchers. The cost is $1,650 CAD (approx. $1,200 USD) for industry affiliated individuals.
Industry registration/supporting
$2,000 CAD (approx. $1,450 USD) support would provide one FREE registration as well as logo and acknowledgement on all event material and website. Please contact us if your organization wishes to send several individuals as we would offer a reduced rate, however, please keep in mind that space is quite limited and we wish to provide access to a wider range of groups.
Registration Process:
Due to limited spots, please express your interest to register (please find the button below) by September 1st, 2024. Notification will be sent out by no later than September 2nd, 2024 with a link to register and pay your registration. You then have one week to pay your registration fee otherwise we will remove your spot and offer it to another participant in the waiting list.
We prioritize diversity in geography, industry, and experience to ensure a rich and inclusive learning environment. For those with financial need, we have a limited number of bursaries available to help offset the cost of attendance. Please send us a note.
Accommodation:
Participants are expected to arrange their own accommodations. There are many hotels and the location is one of the most accessible transportation hubs in Canada with subway, train and bus services within 5 min walk.
Canada VISA:
We are able to request Canada visa invitation letters for registered paid attendees. Once the letter request is approved, we will share the letter asap with you. Please indicate if you will need a visa invitation letter when registering. Please check the Canada embassy in your country as nationals from many countries might not require one.