In this blog, we’ll explore how to create a web application using Replit AI Agent, an exciting tool that can turn your ideas into working applications with just the help of clear prompts. Wondering what a Replit Agent is and how the whole process works? Let’s break it down! 

What is Replit Agent? 

The Replit Agent is an AI-powered development assistant that can build complete applications based on simple, well-written prompts from users. It supports multiple programming languages, but by default, the Replit Agent is most comfortable using: 

• React for front-end development.

• Express.js for the backend.

• PostgreSQL for the database. 

With just a clear set of instructions, the Replit Agent can set up both the front-end and back-end, creating a solid foundation for modern web applications. 

What Does It Take to Build an Application Using Replit Agent? 

At the heart of any application built with Replit Agent lies one key element: the prompt. 

A good prompt is like a blueprint. It should clearly define the features you want, along with any do’s and don’ts, so the AI can understand your exact requirements. 

A few golden rules for writing effective prompts: 

• Avoid overloading your prompt with too many instructions at once. A crowded or overly complex prompt can cause the Agent to stray from your intended outcome. 

• Break your project down into smaller parts and write feature-specific prompts. This helps the Agent focus on one piece at a time, reducing confusion and improving accuracy. 

• Even though the Replit Agent is highly capable, you should always run tests once the feature is generated. Minor bugs or differences from the expected behavior can occur. 

• If you find any bugs or mismatches, you can create a fixing prompt that clearly explains the difference between the actual and expected behavior, attaching a screenshot if possible, can make it even clearer for the Agent. 

For this Timesheet Tracker project, simple Word documents were used to write out each prompt in the same clear structure, adding screenshots whenever available, to guide the Agent. 

A Quick Note on Tech Stack 

By default, Replit Agent uses React with Tailwind CSS for the front-end, and an in-memory database for storing data during development. However, for real-world applications, you’ll likely need a proper database setup. 

To do that, all you need to do is mention it in your prompt. The Agent will automatically: 

• Set up a serverless PostgreSQL database, 

• Create the necessary scripts and schemas, 

• Push the schema to the database, 

• Migrate your data handling from in-memory storage to PostgreSQL. 

You can even query and view your tables directly in the serverless database environment, making it easy to validate and manage your application data. 

Timesheet Tracker Developments using Replit

Prompt 1: The Initial Requirements and Features

When it comes to building an application using Replit AI Agent, everything starts with the right prompt. Think of the prompt as your project’s blueprint, it tells the Agent exactly what you want to build, how it should behave, and what features you expect. 

A clear and structured prompt will make your development smoother and help the AI Agent understand your goals without confusion. 

Here’s what you should keep in mind while writing a prompt, using the Timesheet Tracker example: 

• Clarity is key 
  The prompt should explain the requirements in a straightforward and easy-to-understand way. Each feature, like user roles, timesheet entry, approval flow, and reporting, should be broken down clearly so the Agent knows exactly what to build. 

• Structure your requirements 
  Just like the Timesheet Tracker prompt, group your features logically. For example, User Management, Timesheet Entry, Leave Management, Reporting, and so on. This helps both you and the AI stay organized. 

• Mention region-specific needs 
  If your app has regional differences (like the USA vs. India workweeks or holidays), specify those in the prompt so the Agent can handle them correctly. 

• Role-Based Access is important 
  Define different user roles clearly (Admin, Manager, Employee, HR/Finance) and mention what each role can and cannot do. This avoids confusion later. 

• Don’t forget reports and exports 
  If you need specific reports or formats (Excel, PDF, date ranges, project-wise, etc.) — mention them upfront. This will help the Agent include these features from the start. 

• User-friendly notifications 
  Features like daily or weekly reminders and approval alerts should be included if you want smooth user interactions. 

• Real-world workflows matter 
  Including approval steps, comments, and reminders ensures your app feels natural and helpful for real users, not just technically functional. 

A Few Extra Tips for Writing a Good Prompt: 

• Keep it detailed but easy to follow, avoid overloading the prompt with too much in one go. 

• When possible, split large features into separate prompts, this keeps the AI Agent focused. 

• Include expected behaviors, user scenarios, or screenshots to give clear context. 

• Be specific about tools and tech stack, like telling the Agent to switch from in-memory to Postgres. 

• Always test the results, even when the Agent completes the build — minor bugs or misinterpretations can happen, and follow-up prompts help fix them. 

A well-crafted initial prompt is the secret to making Replit Agent work for you, not against you. The more precise you are, the closer the Agent’s output will match your vision. 

Below is the screenshot of Timesheet Tracker Application Developed by Replit Agent. 

Prompt 2: Bug Identification and Fixing with Replit Agent 

Even though Replit AI Agent does an excellent job generating full-featured applications from well-structured prompts, like any development process, it’s common to encounter minor bugs or unexpected behaviors. The key is to treat these moments as part of the iterative process. 

Once the initial version of the Timesheet Tracker Application was generated, real-time testing was conducted to make sure all the core features worked as intended. During this phase, a set of common functional bugs was identified which were compiled and addressed by sending the Agent new targeted prompts, clearly mentioning: 

• The Current Behavior (what’s happening).

• The Expected Behavior (what should happen).

• Screenshots (if available) for extra context. 

Here’s a quick snapshot of the Bug List 1 encountered: 

• Bug 1: Timesheet Approval/Rejection Error 
  When a Manager clicks “Confirm Approval/Rejection,” an error occurs — 
  value.toISOString is not a function. 
  → The timesheet should approve successfully and show in the approved list. 

• Bug 2: Adding Team Members to a Project 
  Adding team members accidentally adds the logged-in user instead of the selected one and even creates duplicates. 
  → It should correctly assign only the chosen team member to the project. 

• Bug 3: Timesheet Daily/Weekly Toggle Not Working 
  The toggle between Daily and Weekly views has no effect on the display. 
  → Switching the view should update the timesheet display accordingly. 

• Bug 4: Missing Auto-Refresh in Tables 
  After creating a new entry (timesheet, project, task, user, holiday), the new record doesn’t appear unless the page is manually refreshed. 
  → Newly added records should appear automatically in the table. 

• Bug 5: Incorrect Approval Count Displayed 
  The side menu always shows “3” for pending approvals, even if the actual number is different. 
  → The count should reflect the real number of pending approval requests. 

• Bug 6: View Full Calendar Not Working 
  Clicking “View Full Calendar” results in no action and no calendar is displayed. 
  → The app should show a full calendar with region-specific holidays configured by the Admin. 

Once these bugs were documented, Fixing prompts were used to guide the Agent with clear instructions on what went wrong and what the expected behavior should be. Adding screenshots wherever possible helped the Agent better understand UI-related issues, too. 

This approach — testing, identifying, and fixing bugs through follow-up prompts, highlights the importance of combining human testing with AI-generated builds. While the Replit Agent can save a huge amount of development time, it still benefits from human eyes to ensure everything works just the way you intended. 

Checklist – Key Steps to Build with Replit Agent

• Write clear, structured prompts. 

• Split complex features into separate prompts. 

• Test every feature thoroughly. 

• Use fixing prompts to address bugs. 

• Validate and deploy. 

Conclusion 

Building applications with the Replit AI Agent opens up an entirely new way of approaching software development. One where clear communication, through well-structured prompts, becomes just as important as coding itself. 

Throughout this Timesheet Tracker project, you’ve seen how an idea can turn into a working application simply by guiding the Replit Agent with clear instructions and thoughtful iteration. From defining your core functional requirements to identifying and fixing real-world bugs, the development process with AI still mirrors traditional software development in many ways, especially when it comes to testing and refining. 

The biggest takeaway? 

AI like Replit Agent isn’t here to replace developers, it’s here to accelerate your journey from concept to completion. The quality of the output depends entirely on the clarity of your inputs. If you learn to write structured, specific, and scenario-based prompts, you can dramatically reduce development time while maintaining the flexibility to adjust and correct issues as they arise.

So, whether you’re an experienced developer or someone experimenting with AI-powered app building for the first time, Replit Agent is a valuable companion that can help transform ideas into real, testable, and deployable applications, faster than ever. 

In the rapidly evolving business landscape, the competition for top talent has never been fiercer. Traditional hiring methods, such as manual resume screening, repetitive administrative tasks, and multiple rounds of interviews, are increasingly seen as inefficient and outdated.

Enter AI-based talent recruitment and HR automation technologies that are reshaping how organizations attract, engage, and hire talent. These smart solutions enable companies to streamline recruitment, make data-driven decisions, and create fairer and more inclusive workplaces.

How AI is Transforming Talent Recruitment

Artificial Intelligence (AI) is actively reinventing the recruitment process. Here’s how AI enhances talent acquisition:

1. Resume Screening at Scale

AI-powered platforms can analyze thousands of resumes in minutes. They automatically filter candidates based on job descriptions, skills, and experience, significantly cutting down the screening time, sometimes by up to 75%.

2. Predictive Hiring Analytics

Historical hiring data fuels predictive models that help identify candidates most likely to succeed in specific roles. Organizations can make smarter, more informed hiring decisions using these insights.

3. Bias Reduction

Properly designed AI minimizes unconscious bias by focusing on objective qualifications rather than subjective factors, supporting diversity and inclusion efforts.

4. Chatbots for Candidate Engagement

AI-driven chatbots interact with candidates 24/7, answering questions, guiding them through applications, and scheduling interviews, greatly improving the overall candidate experience.

Key Benefits of AI in HR and Recruitment

The integration of AI into HR processes provides numerous advantages:

1. Speed and Efficiency

AI automates repetitive tasks like candidate screening and interview scheduling, allowing HR professionals to focus on strategic initiatives.

2. Improved Candidate Quality

AI matches candidates not only to job descriptions but also considers cultural fit and future career paths, improving retention and performance.

3. Cost Reduction

Organizations save costs by minimizing dependency on agencies and reducing time-to-hire.

4. Enhanced Candidate Experience

Faster application processes, timely feedback, and personalized interactions enhance employer branding.

5. Data-Driven Decision Making

AI-driven insights into sourcing channels, hiring timelines, and employee turnover empower HR leaders to optimize recruitment strategies.

Popular Use Cases of AI in HR

AI applications in HR include:

• Resume screening and ranking.
• Passive candidate sourcing.
• Interview scheduling automation.
• Streamlined employee onboarding.
• Predicting employee performance.
• Identifying attrition risks.

Real-time Examples: Naukri and LinkedIn

Naukri AI Integration

Naukri has started integrating AI mainly for resume screening, job matching, and candidate recommendations. Their AI models help recruiters find better fits faster by analyzing profiles, predicting candidate suitability, and even automating follow-ups. They’re also using AI chatbots to assist users during job applications.

LinkedIn AI Integration

LinkedIn is heavily into AI; they use it for personalized job recommendations, skill assessments, profile optimization tips, and content suggestions. Their AI also powers features like Recruiter AI (which helps companies find candidates smarter) and Career Explorer (which suggests career paths based on your skills).

Challenges and Considerations

Despite its benefits, AI in HR presents challenges:

• Bias in algorithms due to flawed training data.
• Privacy and data security concerns.
• Loss of human touch in recruitment.
• Transparency and fairness of AI decisions.

How to Successfully Implement AI in HR

To implement AI successfully in HR, consider the following steps:

1. Define Clear Goals

Identify pain points and set specific, measurable objectives.

2. Choose the Right Tools

Select AI tools that integrate with existing HR systems, are user-friendly, secure, and proven effective.

3. Monitor and Train Regularly

Continuously evaluate AI system performance and retrain models with fresh data to ensure fairness.

4. Keep the Human in the Loop

Maintain human oversight for final hiring decisions to preserve empathy and judgment.

5. Educate Your Team

Provide training on AI usage, ethics, and responsible application of AI-driven insights.

Future Trends: What’s Next for AI in HR?

Emerging trends include:

• AI-Generated Personalized Learning Paths.
• Emotional AI measuring soft skills.
• Blockchain for secure credentials.
• Hyper-personalized employee experiences.

Conclusion

AI-based recruitment and HR automation are not futuristic concepts — they are today’s strategic advantages. Organizations that harness these technologies thoughtfully will shape a more agile, inclusive, and successful workforce for the future.

In today’s fast-evolving digital landscape, creating seamless and user-friendly experiences is at the heart of product and service design. To achieve this, understanding and optimizing user experience (UX) is crucial. Traditionally, UX testing and feedback analysis have been manual, time-consuming processes that rely heavily on human interpretation. However, the integration of artificial intelligence (AI) is reshaping the UX landscape, making these processes faster, more efficient, and increasingly insightful. Here’s how:

1. Automating UX Testing

AI-powered tools are now capable of automating the majority of UX testing processes. These tools can simulate user interactions, identify usability issues, and provide actionable insights. For example:

• Heatmaps and Click Tracking: AI analyzes where users click or hover the most, predicting behavior and identifying navigation bottlenecks.
• A/B Testing at Scale: AI can run simultaneous A/B tests, optimizing different designs or layouts to determine the best-performing option in real time.

2. AI-Driven Session Replay Analysis

Instead of manually watching hours of screen recordings, AI can cluster similar sessions and detect anomalies. For example, if users often abandon a checkout page, AI can spot the pattern and even identify the potential cause.

3. User Behavior Predictions

Machine learning models learn from past user behaviors to predict future trends. This enables designers to anticipate user needs and proactively refine UX elements. For instance:

• Predicting which features users might find most useful.
• Identifying potential churn risks based on interaction patterns.

4. Accessibility Improvements

AI is also playing a pivotal role in making digital products accessible to all users, including those with disabilities. For instance:

• Automated suggestions for color contrast improvements.
• Voice-to-text and text-to-voice functionalities for inclusivity.

5. Real-time Feedback Loop

One of AI’s most significant contributions to UX is enabling real-time feedback loops. Chatbots and AI assistants embedded in apps or websites collect feedback on-the-go, identifying and addressing issues instantaneously.

6. Reducing Human Bias

Human bias can often affect the interpretation of UX data. AI, when designed carefully, provides an objective analysis, ensuring that decisions are data-driven and user-centric.

Smarter Feedback Analysis with AI

User feedback from app store reviews to survey responses is a goldmine of insights. But when you’re dealing with thousands of comments, it’s nearly impossible to analyze them manually. AI makes feedback analysis faster and smarter.

1. Sentiment Analysis

AI uses Natural Language Processing (NLP) to analyze sentiment in user reviews and support tickets. It can:

• Categorize feedback as positive, negative, or neutral.
• Identify the emotional tone (frustration, delight, confusion).
• Track sentiment trends over time.

This helps teams quickly gauge user satisfaction and identify pain points.

2. Feedback Clustering & Theme Detection

AI can automatically cluster similar feedback and extract common themes like “slow load time,” “confusing onboarding,” or “excellent support.” This eliminates the need for manual tagging and prioritizes issues based on frequency and severity.

3. Voice of Customer (VoC) Insights

Combining sentiment, intent, and behavior analysis, AI delivers deeper Voice of Customer insights. It can even flag feedback from high-value users or influencers, helping teams prioritize responses.

Tools and Technologies Leading the Charge

Some popular AI-driven tools that are transforming UX testing and feedback analysis include:

• Hotjar AI – Smart analysis of heatmaps and user sessions.
• Qualtrics XM – Predictive models for customer experience.
• Monkey Learn – AI for text analysis and feedback tagging.
• User Testing with AI – AI-driven insights from user test videos.
• Chatter mill – Unified AI platform for customer feedback analysis.

Benefits of AI in UX Testing and Feedback

• Speed: Automate testing and analysis that would take weeks manually.
• Scalability: Analyze millions of user sessions or reviews effortlessly.
• Accuracy: Minimize human bias and error in interpreting feedback.
• Proactive Fixes: Predict UX issues before they escalate.
• User-Centric Design: Make data-driven improvements based on real behavior and sentiment.

Conclusion

By automating tedious tasks, uncovering hidden insights, and predicting user behavior, AI helps teams build better products, faster. As digital experiences become more complex, AI will be essential in crafting seamless, user-friendly interfaces that delight and retain users.

So, if you’re in UX design, product management, or QA, it’s time to embrace the AI wave and reimagine how user experience can be understood and improved.

As part of a modernization initiative, Amazon Q Developer, a next-generation developer assistant, has actively been leveraged to accelerate the migration of legacy .NET Framework applications to the latest .NET platforms, including .NET 6, .NET 7, and .NET 8. Amazon Q Developer is designed to support a broad spectrum of application types, making it a highly effective solution for transforming large-scale enterprise portfolios that span multiple layers of architecture and technology stacks.

Understanding Legacy .NET Framework Applications

Legacy .NET Framework applications are those built on older versions of the .NET Framework, such as .NET Framework 3.5 and legacy .NET Core versions. These applications often face limitations in terms of performance, security, and scalability, making modernization essential for maintaining operational efficiency and agility.

Benefits of Migrating to Modern .NET Platforms

Migrating to modern .NET platforms, such as .NET 8, offers several advantages:

• Improved Performance: Modern .NET platforms provide better performance and resource management.
• Cross-Platform Support: Applications can run on various operating systems, including Windows, Linux, and macOS.
• Enhanced Security: Modern .NET platforms offer improved security features and long-term support (LTS).
• Operational Efficiency: Modernization leads to operational efficiency and agility, enabling businesses to adapt to changing market demands.

Amazon Q Developer

Amazon Q Developer is a powerful tool designed to support the migration of legacy .NET Framework applications to modern .NET platforms. It supports a wide range of source project types, ensuring compatibility with even significantly aged applications. The tool provides a safe and structured code transformation process, pushing the transformed code into a separate branch to maintain the integrity of the original repository.

Migration Process

The migration process using Amazon Q Developer involves several steps:

1. Assessing the Legacy Application: Identify the project types and assess the compatibility with Amazon Q Developer.
2. Setting Up Amazon Q Developer: Configure the migration settings and prepare the tool for transformation.
3. Performing Code Transformation: Execute the code transformation and push the transformed code into a separate branch.

Transformation Steps in IDE:

1. Open any C# based solution or project in Visual Studio that you want to transform.

2. Open any C# code file in the editor.

3. Choose Solution Explorer.

4. From the Solution Explorer, right click a solution or project you want to transform, and then choose Port with Amazon Q Developer.

5. The Port with Amazon Q Developer window appears.

The solution or project you selected will be chosen in the Choose a solution or project to transform dropdown menu. You can expand the menu to choose a different solution or project to transform.

In the Choose a .NET target dropdown menu, choose the .NET version you want to upgrade to.

6. Choose Confirm to begin the transformation.

7. Amazon Q begins transforming your code. You can view the transformation plan it generates for details about how it will transform your application.

A Transformation Hub opens where you can monitor progress for the duration of the transformation. After Amazon Q has completed the Awaiting job transformation startup step, you can navigate away from the project or solution for the duration of the transformation.

8. After the transformation is complete, navigate to the Transformation Hub and choose View diffs to review the proposed changes from Amazon Q in a diff view.

9. Choose View code transformation summary for details about the changes Amazon Q made. You can also download the transformation summary by choosing Download summary as .md.

If any of the items in the Code groups table require input under the Linux porting status, you must manually update some files to run your application on Linux.

1. From the Actions dropdown menu, choose Download Linux readiness report.
2. A .csv file opens with any changes to your project or solution that you must complete before your application is Linux compatible. It includes the project and file that need to be updated, a description of the item to be updated, and an explanation of the issue. Use the Recommendation column for ideas on how to address a Linux readiness issue.
3. To update your files in place, choose Accept changes from the Actions dropdown menu.

4. Reviewing and Testing: Review the transformed code, test its functionality, and make necessary adjustments.

Key Features and Capabilities

Amazon Q Developer offers several key features that make the migration process efficient:

• Support for Legacy Project Types: Compatible with a wide range of source project types, including Web APIs, Libraries, Unit Tests, WCF, MVC and SPA.
• Safe and Structured Code Transformation: Ensures the original repository remains untouched by pushing the transformed code into a separate branch.
• Bulk transformation: Amazon Q Developer Web experience provides the ability to transform multiple projects in a single job from external repositories like GitHub and GitLab, without affecting the original source code. The transformation occurs in a newly created branch, ensuring the safety of the original source code.

• Developer-Friendly Review and Control: Supports bulk transformation, generates necessary .NET 8 files, and integrates with IDEs like Visual Studio and VS Code for developer control.

Transformation Use Cases

Amazon Q Developer supports the transformation of various legacy application types, such as:

• Web API Projects: Full transformation to .NET 8 with clean, minimal, and production-ready code. Some manual updates required for ApiController attributes and route mappings.
• MVC Applications: Smooth migration of controllers and back-end logic layers, with manual adjustments needed for the UI layer and Razor views.
• Class Libraries: Seamless migration with accurate updates to project format and references.
• Unit Test Projects: Successful transformation to .NET 8 with validated test execution.

Key Findings and Observations

During the modernization process, we observed the following:

• Web API Projects: Clean and minimal code with some manual updates required.
• MVC Applications: Smooth migration with manual adjustments needed for the UI layer.
• Class Libraries: Accurate updates to project format and references.
• Unit Test Projects: Validated test execution with minimal alteration.
• Projects with External DLL Dependencies: In certain scenarios, a new class is created that references external DLLs or private NuGet packages. It is essential to carefully review and apply changes to ensure compatibility.
• HTTP Attribute Mismatch: REST API best practices for HTTP attributes are applied during transformation, but the original source may not have followed these practices. This can lead to potential compatibility issues for API consumers.       

Best Practices for a Successful Migration

To ensure a smooth and successful migration process, consider the following best practices:

• Thorough Assessment: Assess the legacy application and identify compatibility with Amazon Q Developer.
• Configuration and Setup: Properly configure the migration settings and prepare the tool for transformation.
• Review and Testing: Review the transformed code, test its functionality, and make necessary adjustments.
• Manual Adjustments: Be prepared to make manual adjustments for specific project types and dependencies.

Conclusion

Migrating legacy .NET Framework applications to modern .NET platforms using Amazon Q Developer offers several benefits, including improved performance, cross-platform support, enhanced security, and operational efficiency. The tool provides a safe and structured code transformation process, ensuring compatibility with even significantly aged applications.

Front-end development often focuses on design systems, data binding, and UI responsiveness. But with artificial intelligence (AI) becoming increasingly accessible, it’s time to explore how AI-powered automation can enhance Angular applications.

Automation is no longer confined to the backend. With modern APIs and cloud-based intelligence, Angular developers can now create front-end applications that predict, personalize, and even automate user interactions. This blog explores how AI and automation can be integrated into Angular apps to deliver smarter, more intuitive user experiences.

The Shift: From Interactive to Intelligent

Traditional web apps rely on user input and manual data. Smart apps anticipate needs, adapt to context, and automate tasks. AI bridges that gap.

In Angular apps, this means:

• Predictive suggestions (e.g., auto-completed forms based on input context)
• Automated workflows (e.g., booking systems that fill details intelligently)
• Conversational UIs (e.g., chatbots for onboarding or support)
• Smart content filtering (e.g., displaying content based on sentiment or preferences)

Use Case: Building a Smart Symptom Checker Form in Angular

Scenario:
A healthcare web app requires users to describe symptoms in order to book an appointment. Instead of selecting a specialty and time manually, this process can be automated based on the user’s input.

Solution:
We can use the OpenAI GPT API to classify the symptoms and automatically suggest:

• Relevant medical specialty
• Available doctors
• Suggested appointment time slots

Workflow:
1.  The user enters symptoms in a text box.
2. The input is sent to OpenAI via an Angular service.
3. The API returns a structured interpretation.
4. Form fields are auto-populated with suggestions.

Code Snippet: Angular + OpenAI Integration

@Injectable({ providedIn: 'root' })
export class AIService {
  constructor(private http: HttpClient) {}

  getDiagnosis(input: string): Observable<any> {
    const headers = new HttpHeaders({
      'Authorization': `Bearer YOUR_OPENAI_API_KEY`,
      'Content-Type': 'application/json'
    });

    const body = {
      model: 'gpt-4',
      messages: [{ role: 'user', content: `Suggest a medical specialty for: ${input}` }]
    };

    return this.http.post('https://api.openai.com/v1/chat/completions', body, { headers });
  }
}

Tools to Boost Automation in Angular

Here’s a quick list of AI and automation tools Angular developers can leverage:

Other Real-World AI Automation Ideas for Angular Devs

• Smart HR Portal: Auto-summarize resumes using AI and recommend roles.
• IT Helpdesk Assistant: Triage tickets based on urgency/sentiment.
• AI-Powered Dashboard: Show real-time insights and alerts based on historical data.
• Voice-Controlled Interfaces: Combine Web Speech API + Angular to trigger app actions.
• Personalized Content Feed: Filter content dynamically using user sentiment or interests.

The Future: Angular Meets AI-First Development

With Angular’s evolving architecture and growing support for reactive paradigms (like Signals), it’s becoming easier to integrate real-time data and reactive AI behavior.

As AI becomes mainstream in SaaS products, developers who know how to blend automation and intelligence into their apps will lead the next wave of innovation. Whether it’s through smart forms, personalized dashboards, or conversational UIs, Angular developers have all the tools they need to make the leap.

Conclusion

The future of the front-end isn’t just responsive; it’s intelligent. By integrating AI-powered automation into Angular apps, developers can craft user experiences that are faster, smarter, and more human-centric. No deep ML expertise is required to begin. Start small, experiment with APIs, and gradually build forward-thinking features. Angular is ready for the AI-first era.

💡Tip: Start by automating one small user pain point. Let AI handle the rest.

Product recommendations have come a long way from static, one-size-fits-all suggestions to dynamic, AI-driven personalization. In the early days, businesses used manual curation or simple algorithms that grouped users based on shared behaviors. However, with the rise of big data and machine learning, recommendations have become smarter, faster, and more relevant.

Today, AI-powered engines analyze browsing history, purchase patterns, and even real-time interactions to predict what users want before they even search for it. From e-commerce and streaming to finance and healthcare, personalized recommendations have transformed how businesses engage customers, making interactions seamless, intuitive, and highly effective.

Logic behind AI recommendations

AI-powered engines analyze vast amounts of data to predict what customers may want based on past behaviors, preferences, and trends. Here’s how they work:

How AI Understands Customer Data

AI systems use sophisticated methods to process and interpret multiple data points, delivering personalized suggestions:

Different Types of AI-Powered Recommendation Systems

Collaborative Filtering

Collaborative filtering operates on the principle that users with similar behaviors will have similar preferences. It analyzes past behaviors and interactions of different users to suggest items that others with comparable interests have liked. The system can be either user-based or item-based, depending on the focus of the algorithm.

Content-Based Filtering

This approach recommends products based on the attributes of items a user has previously engaged with. For instance, if a customer frequently purchases sports shoes, the system will suggest other products with similar characteristics, such as running gear or fitness trackers.

Hybrid Recommendation Systems

A combination of collaborative and content-based filtering, hybrid models offer more accurate suggestions by leveraging the strengths of both methods. Netflix, for example, uses a hybrid approach, analyzing viewing history, genres, and ratings to recommend content tailored to each user.

Latest Advancements in AI Recommendations

• Generative AI for Personalized Shopping: AI is now capable of generating hyper-personalized shopping experiences by understanding consumer intent through natural language processing (NLP).
• Large Language Models (LLMs): Retailers are integrating LLMs like GPT to enhance customer interactions, making product discovery more engaging.
• AI Chatbots & Voice Assistants: These tools assist shoppers in finding products by answering queries, making recommendations, and even processing transactions in real-time.
• Computer Vision: AI-powered image recognition allows users to upload pictures to find similar products, enhancing convenience in online shopping.
• Context-Aware Recommendations: AI is evolving to analyze external factors such as location, time of day, and weather to refine its suggestions.

Why AI Recommendations Matter to Consumers and Businesses

AI-powered recommendations bridge the gap between businesses and consumers, creating a win-win situation. They enhance user experiences by delivering relevant suggestions while helping businesses drive engagement, sales, and loyalty.

Benefits for Consumers

• Personalized Shopping Experience: AI ensures that consumers receive product recommendations tailored to their preferences, saving them time and effort.
• Enhanced Convenience: AI-powered recommendations allow users to quickly find relevant products, making online shopping smoother and more efficient.
• Improved Decision-Making: By analyzing trends and previous interactions, AI helps users discover products they may not have otherwise considered.
• Higher Satisfaction: AI-driven recommendations often lead to more satisfying purchases as they align closely with the customer’s needs and preferences.
• Better Content Discovery: On streaming platforms and e-books, AI assists in surfacing content that matches a user’s viewing or reading habits, enhancing entertainment experiences.

Benefits for Businesses

• Increased Customer Engagement: AI-driven recommendations keep customers engaged by showing them relevant products, leading to higher interaction rates.
• Boost in Sales & Revenue: Businesses using AI recommendation engines have reported up to a 50% increase in revenue (Source: Harvard Business Review, 2023).
• Higher Conversion Rates: AI-powered product suggestions have been shown to increase conversion rates by up to 20% (Source: McKinsey & Company, 2022).
• Customer Retention & Loyalty: Personalized recommendations improve customer retention by 30%, as consumers appreciate platforms that cater to their unique preferences (Source: Gartner, 2023).
• Optimized Marketing Strategies: AI can predict which products customers are likely to buy next, allowing businesses to create targeted campaigns that maximize ROI.
• Reduced Cart Abandonment: Personalized recommendations at checkout encourage customers to complete their purchases, minimizing lost sales.
• Competitive Advantage: Companies leveraging AI for recommendations stay ahead by delivering superior user experiences compared to businesses relying on traditional methods.

AI-driven recommendation systems use machine learning and data analytics to provide customers with personalized product suggestions based on their immediate preferences and behaviors. This approach not only enhances customer satisfaction but also significantly impacts revenue by increasing the average order value and improving customer retention.

Statistics Highlighting the Impact

The impact of AI-powered recommendations is backed by data, proving just how essential they are for businesses and consumers alike. Here are some key statistics highlighting their effectiveness:

Market Growth: The product recommendation engine market is projected to grow from $7.42 billion in 2024 to $10.13 billion in 2025, with a compound annual growth rate (CAGR) of 36.5% (The Business Research Company).

Future Projections: By 2029, the market is expected to reach $34.77 billion, driven by increased demand for real-time and personalized shopping experiences (The Business Research Company).

Adoption Rate: Approximately 70% of companies are either implementing or developing digital transformation strategies, which include the use of recommendation engines (ZDNet, cited in Mordor Intelligence).

Impact on Sales: Product recommendations account for 35% of Amazon’s sales, highlighting their significant impact on revenue (Involve.me).

Consumer Preference: 83% of customers are willing to share their data for a more personalized shopping experience (Involve.me).

Conversion Rates: 49% of online purchases are made by consumers who did not intend to buy until they received personalized product recommendations (Digital Minds BPO).

Future Trends in AI Recommendations

1. Hyper-Personalization: AI is moving towards offering deeply personalized experiences by analyzing micro-interactions and emotional responses.
2. Integration with Augmented Reality (AR) & Virtual Reality (VR): AI-powered recommendations will merge with AR/VR to create immersive shopping experiences.
3. Cross-Industry Adoption: Beyond retail, AI recommendations are being integrated into the healthcare, finance, and education sectors to enhance user engagement.
4. Explainable AI (XAI): Researchers are working on making AI recommendations more transparent and interpretable to improve trust and user adoption.

Optimize Product Recommendations with CloudIQ Solutions

To maximize the potential of AI-powered recommendations, businesses need sophisticated, data-driven solutions. CloudIQ Solutions provides state-of-the-art AI recommendation engines designed to enhance personalization, increase engagement, and drive revenue growth. 

By leveraging advanced machine learning and deep learning models, CloudIQ helps businesses deliver tailored product suggestions that keep customers engaged and coming back for more. Stay ahead of the competition with CloudIQ’s intelligent recommendation technology.

Writing robust unit tests can often feel like a chore, especially when working with complex services like AI layers or cloud databases. But what if your AI-powered coding assistant could do most of the heavy lifting for you? Enter GitHub Copilot. In this blog post, we’ll explore how you can use Copilot to supercharge your .NET unit testing efforts, particularly when working with mocked services like AI layers, Cosmos DB repositories, and API integration testing. 

1. Mocking the AI Service Layer for Unit Tests 

Let’s say you’re testing a service that relies on an AI component (like sentiment analysis). Here’s how you can leverage Copilot to mock it efficiently. 

What You Need: 

• An IAIService interface (e.g., Sentiment Analysis) 

• A unit testing framework like xUnit, NUnit, or MSTest 

• A mocking library such as Moq 

 
Copilot prompt:  Write a unit test that mocks IAIService and tests ReviewService.AnalyzeReviewAsync.

Example: 

[Fact] 

public async Task AnalyzeReviewAsync_ShouldReturnEnrichedReview() 

{ 

    // Arrange 

    var mockAIService = new Mock<IAIService>(); 

    mockAIService.Setup(x => x.AnalyzeSentimentAsync(It.IsAny<string>())) 

        .ReturnsAsync(new SentimentResult { Sentiment = "positive" }); 

 

    var reviewService = new ReviewService(mockAIService.Object, ...); 

 

    // Act 

    var result = await reviewService.AnalyzeReviewAsync("Great product!"); 

 

    // Assert 

    Assert.Equal("positive", result.Sentiment); 

} 

2. Testing Cosmos DB Repositories with In-Memory Emulators 

Cosmos DB can be tricky to test, but Copilot helps simplify the setup, whether you’re using the Cosmos DB Emulator or mocking the CosmosClient. 

What You Need: 

• Cosmos DB Emulator or a mocked CosmosClient 

• A repository pattern to abstract the data layer 

• Test setup/teardown logic for clean test states 

Copilot prompt:  Write a unit test for ReviewRepository using a mocked CosmosClient.

Example: 

[Fact] 

public async Task AddReviewAsync_ShouldStoreReviewInDatabase() 

{ 

    // Arrange 

    var mockContainer = new Mock<Container>(); 

    var mockClient = new Mock<ICosmosDbService>(); 

    mockClient.Setup(x => x.AddReviewAsync(It.IsAny<Review>())) 

              .ReturnsAsync(new Review { Id = "123", Sentiment = "positive" }); 

 

    var repo = new ReviewRepository(mockClient.Object); 

 

    // Act 

    var result = await repo.AddReviewAsync(new Review()); 

 

    // Assert 

    Assert.Equal("123", result.Id); 

} 

This helps generate the full test structure, including initialization, mock setup, and expected assertions. 

3. API Testing with Mock Cosmos Responses + Sample AI Outputs 

For end-to-end testing, especially with complex dependency chains, Copilot makes integration testing smooth. 

What You Need: 

• ASP.NET Core integration tests using WebApplicationFactory<> 

• Mocked Cosmos and AI services injected via TestStartup.cs 

• Dependency injection is managed using mock services. 

Copilot prompt:  Write an integration test that posts a review to the API and verifies the response.

[Fact] 

public async Task PostReview_ShouldReturnEnrichedReview() 

{ 

    var client = _factory.CreateClient(); 

 

    var content = new StringContent(JsonConvert.SerializeObject(new ReviewDTO 

    { 

        Text = "Amazing product!" 

    }), Encoding.UTF8, "application/json"); 

 

    var response = await client.PostAsync("/api/review", content); 

 

    response.EnsureSuccessStatusCode(); 

 

    var responseContent = await response.Content.ReadAsStringAsync(); 

    var result = JsonConvert.DeserializeObject<Review>(responseContent); 

 

    Assert.Equal("positive", result.Sentiment); 

} 

Copilot will guide you through mocking services, crafting HTTP requests, and asserting responses. 

Best Practices: Using GitHub Copilot for Unit Testing 

Here are a few pro tips to get the best out of Copilot when writing tests: 

Conclusion 

With Copilot by your side, you no longer need to dread writing tests. It’s like pair programming with an assistant who knows the framework, mocking strategies, and best testing patterns. Whether you’re mocking AI services, emulating Cosmos DB, or writing full-stack integration tests, GitHub Copilot can help turn a time-consuming task into a productive part of development. 

Creating a Web API project from the ground up can often be a time-consuming process, especially when managing multiple layers such as API controllers, business logic, and data handling. With the support of GitHub Copilot, it becomes easier to leverage AI to generate and extend Web API projects efficiently. This article outlines the process of setting up a structured ASP.NET Core Web API project and using GitHub Copilot to add new endpoints quickly. 

Setting Up the Project Structure 

The process begins with creating a new ASP.NET Core Web API project organized with a modular structure for easy maintenance. The following layers are typically included: 

• API Layer: Contains controllers that define API endpoints and handle CRUD operations for product management. 

• Business Layer: Hosts the core business logic, including validation and product-related rules. 

• Data Layer: Includes a class with hardcoded product data for initial testing, eliminating the need for a database during early development. 

• Model Layer: Defines the Product model with properties such as Id, Name, Price, and Description. 

Using GitHub Copilot to Build the API 

Once the foundational structure is in place, GitHub Copilot can be used to streamline the development of project components. Instead of manually writing every part of the application, developers can guide Copilot using tailored prompts. 

For example, after setting up the Model Layer, a prompt such as: 

“Create a controller with endpoints to manage products (CRUD operations) in an API.” 

This enables Copilot to generate an API controller with Create, Read, Update, and Delete endpoints. These endpoints are integrated with the Business Layer and utilize the hardcoded data from the Data Layer. 

Extending the API with New Endpoints 

To expand the API, GitHub Copilot can be prompted while preserving the existing architecture. For instance, to add an endpoint that filters products by price, a prompt like the following can be used: 

Add an endpoint to filter products based on price greater than or less than a given amount, and update the following layers: 

1. API Layer: Add a new endpoint for filtering by price. 

2. Business Layer: Add filtering logic. 

3. Model Layer: Ensure necessary properties exist. 

4. Data Layer: Support filtering in the repository.
    

Copilot can then generate the corresponding code across all relevant layers, adapting to the project’s structure and maintaining consistency. 

Running the Project Locally 

To build and run the project locally, GitHub Copilot can assist with environment setup. Using tools like Visual Studio Code and the .NET CLI, the project can be quickly compiled and executed. After resolving any dependencies and integrating Copilot’s code suggestions, testing can be conducted using tools like Postman to ensure all endpoints function correctly. 

Key Benefits of Using GitHub Copilot 

Implementing GitHub Copilot for Web API development offers multiple advantages: 

• Time Efficiency: Significantly reduces time spent on repetitive coding tasks. 

• Consistency: Helps maintain a uniform coding pattern across different layers. 

• Faster Iterations: Simplifies the process of adding or modifying features. 

Conclusion 

By combining a well-structured ASP.NET Core Web API with GitHub Copilot, teams can accelerate development while maintaining clean, scalable architecture. GitHub Copilot’s ability to understand project context and respond to developer prompts proves highly effective in enhancing productivity. For developers seeking to streamline their Web API workflows, GitHub Copilot offers a practical and efficient AI-powered solution. 

Artificial intelligence is the use of technologies to build machines and computers that can mimic cognitive functions (see images, listen to speech, understand text, make recommendations, etc.) associated with human intelligence.  Machine learning is a subset of AI that lets a machine learn from data without being explicitly programmed. 

Google Cloud Platform (GCP) offers a rich suite of AI and machine learning tools catering to users across different experience levels — from business analysts to seasoned ML engineers. Whether you’re analyzing structured data, classifying images, building custom deep learning models, or tapping into generative AI, there’s a GCP service tailored for you. 

In this expanded guide, you’ll learn: 

• Key AI/ML tools in Google Cloud 

• How to use them from the Cloud Console 

• Practical examples and sample code 

• Use cases, limitations, and best practices 

• A comprehensive summary table to guide your choices 

1. BigQuery ML 

BigQuery ML democratizes machine learning by enabling analysts to build models using standard SQL syntax directly within BigQuery. It’s ideal for use cases involving large, structured datasets — like customer churn prediction, sales forecasting, and classification tasks. 

Key Features 

• Supports classification, regression, time series, and clustering.

• Uses built-in SQL functions for training, evaluation, and prediction.

• No need to move data outside BigQuery.
  

Accessing BigQuery ML 

1. Navigate to BigQuery Studio – https://console.cloud.google.com/bigquery 

2. Open a project and select your dataset. 

3. Use the SQL workspace to run queries like CREATE MODEL or ML.PREDICT. 
   

Sample: Logistic Regression 

CREATE OR REPLACE MODEL `my_dataset.customer_churn_model` 
OPTIONS(model_type='logistic_reg') AS 
SELECT 
  tenure, 
  monthly_charges, 
  contract_type, 
  churn 
FROM 
  `my_dataset.customer_data`;

Best Practices 

• Normalize your numeric features before training.

• Use ML.EVALUATE to assess model performance.

• Partition of large datasets for efficient model training.

2. Vertex AI 

Vertex AI is GCP’s fully managed ML platform that provides a single UI and API for the complete ML lifecycle. It includes support for AutoML, custom model training, pipelines, feature store, and model deployment. 

a) Vertex AI AutoML 

AutoML simplifies model training by abstracting the heavy lifting of data preprocessing, feature selection, and hyperparameter tuning. It supports: 

• Tabular classification/regression 

• Image classification/object detection 

• Text sentiment/entity analysis 

• Video classification 

Accessing AutoML 

1. Go to Vertex AI Dashboard – https://console.cloud.google.com/vertex-ai 

2. Click “Train new model”. 

3. Choose your data type (tabular, image, text, etc.). 

4. Upload or link to your dataset. 

5. Follow the guided training wizard. 

Sample Code: Image AutoML (Python) 

from google.cloud import aiplatform 
 
aiplatform.init(project="your-project", location="us-central1") 
 
dataset = aiplatform.ImageDataset.create( 
    display_name="my-image-dataset", 
    gcs_source=["gs://your-bucket/images/"], 
    import_schema_uri=aiplatform.schema.dataset.ioformat.image.single_label_classification, 
) 
 
job = aiplatform.AutoMLImageTrainingJob( 
    display_name="image-classifier-job", 
    model_type="CLOUD", 
    multi_label=False, 
) 
 
model = job.run( 
    dataset=dataset, 
    model_display_name="my-image-model", 
    training_filter_split={"training_fraction": 0.8, "validation_fraction": 0.1, "test_fraction": 0.1}, 
) 

Limitations 

• Training time depends on data size and complexity.

• Less control over model internals.

b) Vertex AI Custom Training 

Custom training is for advanced users who want to use frameworks like TensorFlow, PyTorch, or XGBoost. You can train models using your own scripts in Docker containers or managed Jupyter environments. 

Accessing Custom Training 

1. In Vertex AI, go to Training → Custom Jobs. 

2. Choose your container or upload a training script. 

3. Specify machine specs (CPUs, GPUs, TPUs). 

Sample Code (Python SDK) 

from google.cloud import aiplatform 
 
aiplatform.init(project="your-project-id", location="us-central1") 
 
job = aiplatform.CustomTrainingJob( 
    display_name="custom-train-job", 
    script_path="train.py", 
    container_uri="gcr.io/cloud-aiplatform/training/tf-cpu.2-11:latest", 
    model_serving_container_image_uri="gcr.io/cloud-aiplatform/prediction/tf2-cpu.2-11:latest" 
) 
 
model = job.run(replica_count=1, machine_type="n1-standard-4") 

Use Cases 

• NLP models using Transformers.

• Computer vision models with custom CNNs.

• Reinforcement learning pipelines.
  

3. Pre-trained APIs 

Google Cloud offers pre-trained APIs that let you access powerful AI capabilities with minimal setup. These are RESTful services available via API calls or SDKs. 

Key Services 

Accessing Pre-trained APIs 

1. Go to API Libaray – https://console.cloud.google.com/apis/library 

2. Enable the required API. 

3. Create credentials (API key or service account). 

4. Use client libraries (Python, Node.js, Java, etc.) or REST calls. 

Sample: Vision API (Label Detection) 

from google.cloud import vision 
 
client = vision.ImageAnnotatorClient() 
 
with open("photo.jpg", "rb") as image_file: 
    content = image_file.read() 
 
image = vision.Image(content=content) 
response = client.label_detection(image=image) 
 
for label in response.label_annotations: 
    print(f"{label.description}: {label.score:.2f}") 

4. Generative AI with Gemini  

Google’s Gemini APIs power generative AI features such as chatbots, summarization, code completion, and document synthesis. These are hosted on Vertex AI with tools like Model Garden and Vertex AI Studio. 

Accessing Generative AI 

1. Visit Vertex AI Studio – https://console.cloud.google.com/vertex-ai/studio 

2. Use a prompt gallery or freeform chat interface. 

3. Choose a language model (Gemini Pro, Gemini Flash, etc.). 

4. For programmatic access, use vertexai Python SDK. 

Sample Code: Text Generation 

Use Cases 

• Automating customer service (chatbots).

• Creative writing or story generation. 

• Code suggestions or bug fixes.

5. Choosing the Right Tool 

6. Conclusion and Resources 

Google Cloud provides one of the most comprehensive, scalable, and user-friendly AI ecosystems available today. With services for every level of expertise, you can start with SQL in BigQuery ML and grow into training deep models in Vertex AI. Pair that with powerful APIs and generative tools — and you have everything you need to build production-ready AI. 

Helpful Links

• Vertex AI Documentation 

• BigQuery ML Documentation 

• AI/ML Product Overview 

• Google Cloud SDK 

Happy experimenting! 

CloudIQ is excited to announce our Sitecore Silver Partnership designation, marking a significant milestone in our commitment to delivering exceptional digital experiences and e-commerce solutions.

This partnership highlights our expertise in Sitecore’s comprehensive suite of products, including Sitecore XP, Sitecore XM Cloud, Sitecore Content Hub, Sitecore CDP, Sitecore Personalize, Sitecore Send, and Sitecore OrderCloud.

Sitecore Solutions Expertise

As a Sitecore Silver Partner, CloudIQ has successfully met Sitecore’s rigorous standards for competency and performance. Our team, consisting of Sitecore-certified MVPs, brings proven expertise, from initial planning to full-scale deployment—ensuring the optimal performance of Sitecore platforms across a wide range of business needs.

Client-Centric Approach

CloudIQ understands the need for cost-effective solutions and a strategic approach. From infrastructure audits to comprehensive Sitecore solution design and implementation, we offer scalable models for onshore and hybrid teams, ensuring flexibility and efficiency in project execution. Our team is committed to driving your digital transformation with Sitecore, offering expert guidance and seamless migrations to the latest versions and Sitecore SaaS offerings.

Commitment to Quality and Customer Success

Achieving Sitecore Silver Partnership highlights CloudIQ’s dedication to excellence. We continue to focus on driving customer success, ensuring that every solution we implement is tailored to meet the unique needs of our clients. From strategy and design to execution and optimization, CloudIQ’s Sitecore expertise ensures that businesses can effectively navigate the complexities of the digital landscape and achieve sustained growth.

Contact us today to learn how CloudIQ’s Sitecore expertise can optimize your digital ecosystem.

For decades, improving patient experience has been a top priority for healthcare systems . Yet, despite these efforts, many healthcare systems still face challenges like long wait times, impersonal interactions, and inefficiencies that hinder optimal care. 

Artificial intelligence has begun to revolutionize healthcare delivery, offering solutions that personalize care, reduce friction in patient interactions, and provide valuable insight. But how deep does AI’s potential go in transforming patient experience? The more we explore, the more we realize that AI isn’t just a tool; it’s a fundamental shift in how healthcare will operate in the years to come. In this blog, we explore AI’s impact on patient interactions, the various AI solutions in use today, and why adoption rates are skyrocketing across the industry.

AI Patient Experience Solutions Delivering Value Across Multiple Use Cases

1. Intelligent Virtual Assistants

Intelligent virtual assistants, often powered by chatbots, are revolutionizing patient engagement by providing immediate responses to inquiries and facilitating appointment scheduling. These AI solutions enhance communication between patients and healthcare providers, ensuring that patients receive timely information about their health needs and appointments. They can also offer personalized health tips based on individual patient data, thereby improving adherence to treatment plans.

2. Remote Patient Monitoring

Remote patient monitoring utilizes AI to continuously track patients’ vital signs and health metrics through wearable devices. This technology allows healthcare providers to receive real-time updates on patient conditions, enabling proactive interventions when necessary. By integrating remote monitoring with telehealth services, patients can maintain regular contact with their healthcare teams without the need for frequent in-person visits.

3. Predictive Analytics

Predictive analytics in healthcare employs AI to analyze historical patient data and predict future health outcomes. This capability allows providers to identify at-risk patients early, facilitating timely preventive measures. By understanding patterns and trends in patient data, healthcare organizations can tailor interventions to improve overall health outcomes and reduce hospital readmissions.

4. Personalized Communication solutions

AI-driven personalized communication tools enhance the way healthcare providers interact with patients. By analyzing individual preferences and behaviors, these solutions can deliver customized messages, reminders for appointments, and educational resources tailored to specific health conditions. This personalized approach fosters better engagement and satisfaction among patients.

5. Automated Administrative Processes

AI is optimizing administrative tasks like scheduling, billing, and claims processing, reducing the administrative burden on healthcare staff. This shift to automation enhances operational efficiency and allows healthcare teams to contribute to a better overall patient experience.

Leading Healthcare Organizations Using Patient Experience Solutions for

1. GetWellNetwork

GetWellNetwork is a leading platform that enhances patient engagement through interactive technology solutions. It provides personalized content and resources tailored to individual patient needs, improving their overall experience during hospital stays. The platform enables patients to access educational materials about their conditions and treatment options while facilitating communication with their care teams.

2. LumaHealth

LumaHealth focuses on improving patient communication through its innovative platform that automates appointment reminders and follow-ups. By utilizing AI-driven messaging systems, LumaHealth ensures that patients receive timely notifications about their appointments and necessary health checks. This proactive approach helps reduce no-show rates and enhances patient adherence to care plans.

3. AthenaHealth

AthenaHealth offers a comprehensive suite of cloud-based services designed to improve practice management and patient engagement. Its AI capabilities include automated appointment scheduling, billing processes, and personalized communication strategies that enhance the overall patient experience. The platform’s focus on interoperability ensures seamless data sharing among providers for coordinated care.

4. Phreesia

Phreesia is an innovative solution that streamlines the check-in process for patients using digital kiosks and mobile applications powered by AI technology. Patients can complete forms electronically, reducing wait times and enhancing operational efficiency for healthcare providers. This solution also allows for personalized messaging based on individual health needs.

5. CipherHealth

CipherHealth specializes in enhancing post-discharge communication through its AI-driven outreach programs. By utilizing automated calls and text messages, CipherHealth ensures that patients receive essential follow-up care instructions after leaving the hospital. This proactive engagement helps improve recovery rates and reduces readmission risks.

AI Adoption and Market Growth in Healthcare Industry

The adoption of AI in healthcare has seen remarkable growth over the past few years. According to recent studies by Dialog Health, approximately 86% of healthcare organizations are investing in AI technologies to improve patient experiences. 

The global market for AI in healthcare is projected to reach $45 billion by 2026, reflecting a compound annual growth rate (CAGR) of over 40% from its 2022 valuation, as reported by Market.us Scoop. These statistics underscore the increasing reliance on AI solutions as healthcare providers seek innovative ways to enhance efficiency while delivering personalized care.
Furthermore, surveys indicate that around 70% of patients express a willingness to use AI-powered digital health solutions for managing their health, according to findings from the Chief Healthcare Executive.com. This growing acceptance among patients highlights the potential for AI technologies to transform traditional healthcare practices into more engaging and effective experiences.

Leading Healthcare Leaders adopted AI-Powered Patient Experience

Our latest findings highlight the top 26 healthcare organizations that are leading the charge in integrating AI into their patient experience strategies. These organizations are leveraging AI to improve patient engagement, reduce wait times, and enhance communication, ultimately providing more efficient and personalized care.

1. Atlantic Health System
2. Banner Health
3. Children’s Healthcare of Atlanta
4. Children’s Mercy Kansas City
5. Grady Health System
6. Guthrie
7. Hackensack Meridian Health
8. Intermountain Health
9. Johns Hopkins Medicine
10. LifeStance Health
11. Maring Health
12. Mile Bluff Medical Center
13. Moffitt Cancer Center
14. Ochsner Health
15. Piedmont
16. Premise Health
17. Reynolds University
18. Saint Luke’s Health System Kansas
19. Sanford Health
20. Southcoast Health
21. Southeast Georgia Health System
22. Trinity Health
23. UC Davis Health
24. Unified Women’s Healthcare
25. WellSpan Health
26. Wellstar Health System

By integrating AI-powered solutions such as virtual assistants, predictive analytics, and automated scheduling, these providers are improving not only operational efficiency but also patient satisfaction.

As AI continues to evolve, its role in healthcare is expanding, offering more precise and personalized care options. These advancements are allowing healthcare professionals to spend less time on administrative tasks and more time focusing on patient needs. With AI-driven tools, providers can anticipate patient needs, ensure timely follow-ups, and enhance the overall healthcare experience.

Interested in improving patient engagement and satisfaction through AI?

Contact us to explore customized solutions for your healthcare organization.

Medical imaging technologies have long been integral to diagnostic medicine, yet interpreting these images demands significant time and expertise. Artificial intelligence (AI) is addressing these challenges by transforming how medical images are analyzed, enabling faster, more precise, and reliable interpretations. This innovation is particularly impactful in fields like radiology, oncology, and neurology, where timely and accurate diagnoses can save lives.

In this blog, we’ll explore how AI is advancing medical imaging, delve into its real-world applications that are helping doctors make better decisions, and examine how AI is being adopted in healthcare systems worldwide. With AI’s growing presence in medical imaging, it’s paving the way for more accurate diagnoses and faster, better care for patients.

Medical Imaging Solutions Delivering Value Across Multiple Use Cases

Here are some of the top AI solutions used in medical imaging, along with their primary use cases:

1. IBM Watson for Oncology

Use Case: Oncology Diagnostics

IBM Watson for Oncology leverages AI to analyze vast amounts of medical data, including clinical literature and patient records, to assist oncologists in making treatment decisions. It provides personalized recommendations based on a patient’s unique profile, enhancing the precision of cancer care.

2. ENDEX by Enlitic

Use Case: General Medical Imaging Analysis

ENDEX utilizes deep learning algorithms to analyze various medical images such as X-rays, CT scans, and MRIs. It detects abnormalities like tumors and fractures with high accuracy, aiding in early diagnosis and treatment planning. Its user-friendly interface facilitates integration into existing workflows, making it accessible to healthcare providers.

3. IDx-DR

Use Case: Ophthalmology

IDx-DR is an FDA-approved autonomous AI system specifically designed for detecting diabetic retinopathy through retinal image analysis. It evaluates images captured by fundus cameras, identifying critical signs of the disease that could lead to blindness if not addressed promptly.

4. Zebra Medical Vision

Use Case: Multi-specialty Imaging Analysis

Zebra Medical Vision offers a suite of AI solutions that analyze medical images across various specialties, including radiology and cardiology. The platform is capable of detecting conditions such as fractures, cardiovascular diseases, and liver conditions from X-rays and CT scans, facilitating timely interventions.

5. Arterys Cardio AI (Tempus Pixel Cardio)

Use Case: Cardiovascular Imaging

This solution automates the analysis of cardiac MRI images using advanced deep learning algorithms. It quantifies cardiac parameters like blood flow and tissue characterization, providing clinicians with valuable insights for diagnosing and managing heart conditions with enhanced accuracy.

6. Siemens Healthineers AI-Rad Companion

Use Case: Radiology Workflow Enhancement

The AI-Rad Companion automates the highlighting and quantification of anatomical structures in imaging studies such as chest CTs. This streamlines the workflow for radiologists by providing automated assessments that reduce interpretation time and improve diagnostic consistency.

7. Blackford

Use Case: Image Reconstruction

Blackford offers AI-powered solutions for medical image reconstruction that enhance detail and reduce noise in CT scans. This improves image quality, which is crucial for accurate diagnosis.

Leading Healthcare Organizations Using Medical Imaging for

1. Enhanced Diagnostic Accuracy

AI-powered solutions excel at identifying patterns and anomalies that might be subtle or overlooked by the human eye. For instance, AI algorithms trained on vast datasets can detect early-stage cancers, cardiovascular irregularities, and other conditions with remarkable precision. This improves diagnostic confidence and reduces the risk of misdiagnosis.

2. Early Detection of Diseases

AI can analyze medical images to detect early signs of diseases before they become symptomatic. This capability allows for the identification of conditions such as cancers, heart disease, and neurological disorders in their earliest stages, when treatment options are often more effective and less invasive. By recognizing subtle patterns that may be missed by the human eye, AI enables timely interventions, improving patient outcomes.

3. Faster Diagnosis and Intervention

Traditional imaging analysis can be time-intensive, particularly in high-volume healthcare settings. AI significantly reduces the time needed to process and interpret imaging results, enabling physicians to provide quicker diagnosis. This is especially critical in emergency situations, such as stroke or trauma, where time is a crucial factor.

4. Personalized Treatment Planning

By analyzing imaging data alongside patient histories and other clinical factors, AI can assist in creating tailored treatment plans. For example, it can predict tumor progression or assess the likely success of a particular therapy, ensuring that treatment is customized to the individual patient’s needs.

5. Improved Workflow and Productivity

AI automates repetitive tasks such as image segmentation, prioritization of urgent cases, and report generation. This allows radiologists and other healthcare professionals to focus on complex cases and patient care, reducing burnout and enhancing overall productivity.

AI Medical Imaging Market Growth

The global AI medical imaging market is projected to grow significantly, from $5.86 billion in 2024 to $20.40 billion by 2029, reflecting a compound annual growth rate (CAGR) of 28.32% (Source: MarketsandMarkets, 2023). This growth is driven by the increasing adoption of AI technologies for disease diagnosis and image analysis, which are enhancing diagnostic accuracy and operational efficiency.

Similarly, the AI diagnostics market is expected to rise from $1.85 billion in 2024 to $14.76 billion by 2034, at a CAGR of 23.1% (Source: Allied Market Research, 2023). This expansion is largely driven by the growing demand for accurate diagnostic solutions and the integration of AI into various diagnostic processes.

Leading Healthcare Leaders adopted AI-Powered Medical Imaging

Our recent research has identified the top 32 healthcare organizations that have successfully integrated AI technologies into their medical imaging practices, setting new standards in diagnostic accuracy, efficiency, and patient care.

The continued adoption of these technologies promises to elevate the quality of care, enabling faster, more precise diagnoses and improving decision-making across various medical specialties. As AI becomes more integrated into medical imaging, it not only enhances diagnostic accuracy but also optimizes workflows, allowing healthcare professionals to focus more on patient care. 

With healthcare systems worldwide embracing AI innovations, patients will benefit from timely, personalized care, while medical professionals gain the solutions needed to deliver better health outcomes. The advancements in AI medical imaging are already making a significant difference in healthcare, with their impact expected to grow in the coming years.

Interested in enhancing your diagnostic processes with AI solutions?

Reach out for more details!

The healthcare industry is experiencing a digital shift, with AI-powered chatbots playing a key role in improving efficiency and patient care. These intelligent assistants streamline operations, support healthcare professionals, and engage patients in real-time.

As AI chatbots gain traction in healthcare, they are driving improvements in everything from patient scheduling to providing personalized medical advice. This blog explores the unique ways AI chatbots are making a difference, the growth in their adoption, and some of the top AI chatbot solutions that are setting new standards in the industry.

Leading Healthcare Organizations using AI-Chatbots for

Symptom Assessment and Triage

AI chatbots analyze patient symptoms, providing initial triage to direct individuals to appropriate care levels, which reduces the burden on emergency departments. Their advanced natural language processing (NLP) capabilities allow them to assess the urgency of symptoms and suggest potential diagnoses, ensuring timely care.

Appointment Scheduling

Chatbots automate the appointment booking process by integrating seamlessly with electronic health record (EHR) systems. This eliminates administrative overhead, reduces scheduling errors, and ensures patients can easily book, reschedule, or cancel appointments.

Medication Reminders

AI chatbots send personalized reminders to patients for medication intake, enhancing adherence to treatment plans and reducing hospital readmissions. This promotes better recovery and overall patient health.

Post-Treatment Follow-Up

After patient discharge, chatbots check in with patients to gather recovery data and alert physicians if intervention is needed. This continuous monitoring improves patient outcomes by ensuring timely follow-up and proactive care.

Health Education

With access to vast medical databases, chatbots provide accurate and easy-to-understand health information, empowering patients to make informed decisions about their care. This helps improve patient education and overall engagement.

24/7 Patient Support

AI chatbots offer round-the-clock assistance, answering questions, providing symptom information, and guiding patients on the next steps. This enhances patient satisfaction by ensuring continuous access to support and timely care.

Scalability Across Healthcare Systems

AI chatbots can be deployed across large healthcare networks, ensuring consistent patient support across multiple locations. Their ability to manage a high volume of interactions simultaneously makes them essential in supporting large patient populations.

AI-powered Chatbot solutions in the market

Ada Health

Ada Health uses AI to guide patients through a personalized diagnostic process. By asking targeted questions and leveraging a symptom checker, it provides instant medical advice based on patient-reported symptoms, seamlessly integrating with healthcare systems to streamline patient interactions.

Buoy Health

Buoy Health’s AI-driven platform assists patients in assessing symptoms and recommending potential diagnoses. It helps alleviate strain on healthcare systems by triaging cases before they reach a physician or clinic.

Woebot

Woebot delivers cognitive behavioral therapy (CBT) to support mental well-being. By tracking mood changes and offering therapeutic conversations, it also directs users to appropriate healthcare resources when necessary.

IBM Watsonx Assistant

IBM Watsonx Assistant aids healthcare providers and patients by answering medical queries, scheduling appointments, and supporting administrative tasks. Its advanced natural language processing ensures accurate responses based on integrated healthcare databases.

Notable Patient AI Platform

Notable’s AI platform enhances patient engagement by automating appointment scheduling, check-ins, and post-visit follow-ups. Integrated with electronic health record (EHR) systems, it streamlines patient-provider interactions.

AI Adoption and Market Growth in Healthcare Industry

The healthcare chatbot market is experiencing rapid growth, with projections indicating an increase from $0.35 billion in 2023 to $0.45 billion in 2024, reflecting a compound annual growth rate (CAGR) of 25.7% (Persistence Market Research, 2024). By 2030, the market is expected to reach $1.18 billion, with a robust CAGR of 27.7% from 2024 to 2028. This growth is fueled by rising healthcare costs, shortages of healthcare professionals, and the increasing demand for immediate access to medical information.

In terms of adoption, approximately 37% of consumers reported using generative AI for health-related purposes as of late 2024 (PYMNTS, 2024). Additionally, about 75% of healthcare leaders are either experimenting with or planning to scale generative AI across their organizations (The Business Research Company, 2024). These statistics highlight the growing recognition of AI chatbots as essential tools in enhancing operational workflows and improving patient engagement within the healthcare sector.

Leading Healthcare Organizations Adopting AI Chatbot Solutions

Our latest research identifies 20 healthcare organizations at the forefront of integrating AI-powered chatbot solutions. These industry leaders are transforming patient experience, offering a new level of efficiency, personalization, and responsiveness in care delivery.

• AccentCare
• Ballad Health
• CommonSpirit Health
• Gillette Children’s Hospital
• Hattiesburg Clinic
• LifeBridge Health
• LifeStance Health
• MemorialCare
• Nemours
• Northwell Health
• Ochsner Lafayette General
• Palomar Health
• Parkview Health
• Saint Luke’s Health System Kansas
• Sharp HealthCare
• Southeast Georgia Health System
• Tidelands Health
• UC Davis Health
• Vituity
• Wellstar Health System

The integration of AI chatbots is transforming how healthcare providers communicate with patients. By automating routine inquiries and streamlining communication, chatbots are enhancing patient access to information, reducing wait times, and supporting clinical workflows. These advancements allow healthcare teams to focus more on direct patient care, while patients receive quicker and more accurate responses to their needs.

As the healthcare sector embraces AI-powered chatbots, the focus shifts to improving patient experience, operational efficiency, and overall care outcomes. These solutions are not only reducing operational costs but also shaping the future of healthcare interactions, making them more accessible and efficient.

Ready to elevate patient experience with AI-driven chatbot solutions?

Contact us to learn more!

In an industry as critical as healthcare, time spent on administrative tasks is time taken away from patient care. With growing patient volumes, rising operational costs, and mounting administrative burdens, traditional documentation methods are becoming increasingly unsustainable. Physicians, under significant pressure, are confronted with time-consuming transcription processes that detract from the quality of patient care.

As the demand for faster, more accurate transcription rises, healthcare organizations are turning to advanced solutions that promise to streamline workflows and alleviate administrative strain. With powerful AI-driven solutions combining automation and intelligent transcription, healthcare providers can significantly reduce time spent on paperwork while enhancing the accuracy and completeness of their medical records. 

This shift is more than a technological upgrade, it represents a necessary transformation that drives operational efficiency, ensures advanced documentation, and ultimately enhances patient care.

Leading Healthcare Organizations using AI-based Transcription

1. Saves Time for Physicians

Clinical documentation significantly reduces the time physicians spend on documentation. Instead of manually typing notes, physicians can dictate their findings during or after consultations. By automating this process, it reduces the hours spent on post-visit paperwork, freeing up physicians to focus on patient care, see more patients, reduce overtime, and maintain a better work-life balance.

2. Improves Documentation Accuracy

With advanced technologies like voice recognition and natural language processing (NLP), clinical documentation ensures that complex medical terms, diagnoses, and abbreviations are accurately captured. This reduces the risk of errors in patient records, which is essential for maintaining quality care, ensuring regulatory compliance.

3. Customization for Medical Specialties

Modern clinical documentation solutions adapt to the specific needs of different specialties, whether it’s cardiology, oncology, or radiology. They support specialty-specific templates, terminology, and formats.Tailored documentation improves the precision and relevance of medical records across various healthcare fields.

4. Cost Savings

By reducing reliance on manual documentation processes and traditional transcription services, clinical documentation minimizes operational costs. Digital solutions eliminate errors, reduce rework, and speed up workflows. These savings allow healthcare facilities to allocate resources more effectively, improving patient services and optimizing operational budgets.

5. Enhances Operational Efficiency

By streamlining the documentation process, clinical documentation improves overall operational efficiency in healthcare settings. Notes are generated faster, post-visit documentation is simplified, and integration with EHR systems eliminates redundant data entry. This optimized workflow enables healthcare teams to function more productively, ultimately improving patient care delivery.

AI-based Transcription solutions in the market

1. CloudIQ Technologies Transcription Solution

CloudIQ’s AI-powered telemedicine application, built on Microsoft Azure Services, uses OpenAI’s Whisper model to transcribe physician-dictated notes in real time and leverages the ChatGPT API to convert them into structured, accurate documentation. By seamlessly integrating into clinical workflows, this solution reduces administrative burdens, saves physicians over 2 hours per day, and enables the treatment of 4,000 additional patients daily, improving productivity and patient care.

2. Dragon Medical One

Dragon Medical One by Nuance is a cloud-based speech recognition software that allows healthcare professionals to dictate directly into healthcare systems. It supports medical terminology and customizable commands for efficient documentation. Its cloud-based architecture facilitates remote access, enhancing productivity across various settings.

3. M*Modal Fluency

3M M*Modal Fluency uses AI to assist with real-time voice recognition and transcription, converting speech into structured clinical documentation. It integrates with EHR systems, streamlining documentation across various specialties and improving accuracy by recognizing medical terminology specific to fields like radiology and cardiology.

4. DeepScribe 

DeepScribe transcribes physician-patient conversations in real time, automatically structuring documentation with minimal input. It integrates into hospital workflows, handles specialized medical terminology, and automates post-encounter documentation.

5. Suki AI

Suki leverages voice recognition for clinical documentation, provides real-time diagnosis code suggestions to assist with coding and billing, and retrieves patient details like medications, allergies, and history, supporting informed decision-making during consultations.

AI Adoption and Market Growth in Healthcare Industry

The global healthcare AI market was valued at USD 19.27 billion in 2023 and is projected to grow at a compound annual growth rate (CAGR) of 38.5% from 2024 to 2030 (Grand View Research). This rapid growth underscores the increasing adoption of AI technologies across healthcare systems.

AI-powered solutions, particularly in clinical documentation, are gaining significant traction. The clinical documentation market alone is expected to grow from USD 2.5 billion in 2024 to USD 6.6 billion by 2031 (Persistence Market Research). These solutions are helping healthcare providers automate documentation processes, save time, and enhance the accuracy of patient records.

In fact, 79% of healthcare organizations have already adopted AI technologies (Microsoft-IDC Study, 2024), reflecting the sector’s significant move toward digital transformation.

Top Healthcare Companies adopted AI clinical documentation

Our latest research highlights 40 leading healthcare organizations that have successfully adopted AI-powered clinical documentation solutions. These innovators are transforming healthcare documentation with cutting-edge technology, setting new standards for efficiency and accuracy in patient care.

The successful adoption of AI clinical documentation tools by these organizations underscores a broader trend in healthcare. As more institutions embrace these technologies, the focus shifts toward improving clinician productivity, enhancing patient care, and ensuring regulatory compliance.

With AI clinical documentation solutions, these organizations are not only increasing efficiency but also paving the way for the next evolution in healthcare. As we move forward, we can expect AI-powered transcription to become an essential part of every healthcare provider’s toolkit.

Looking to reduce administrative burdens in healthcare with AI-driven clinical documentation?

Contact us to get started!

Data has become one of the most valuable assets for businesses, driving the decisions that lead to growth and efficiency. But on its own, raw data can be complex and difficult to interpret. That’s where data analysis comes in. By breaking down information, identifying patterns, and creating visual insights, data analysis makes it possible to transform numbers into clear answers. Microsoft Power BI is one such tool, making this process not only easier but more accessible to users at all levels.

In this blog, we’ll walk through the basics of data analysis and visualization, explore why it’s essential for businesses today, and Power BI empowers organizations to make smarter, data-driven decisions. Whether you’re new to data or just beginning to explore Power BI, this guide will give you the foundational steps to start creating your own data-driven reports.

What is Data Analysis?

Data analysis is the process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making.  In today’s business world, data analysis plays a crucial role in making decisions more scientific and helping businesses operate more effectively.

What is Business Intelligence?

Business intelligence (BI) is a technology-driven process for analyzing data and delivering actionable information that helps executives, managers and workers make informed business decisions. As part of the BI process, organizations collect data from internal IT systems and external sources, prepare it for analysis, run queries against the data and create data visualizations, BI dashboards and reports to make the analytics results available to business users for operational decision-making and strategic planning.

What is Power BI?

Microsoft Power BI is an interactive data visualization software product developed by Microsoft with a primary focus on business intelligence. It is part of the Microsoft Power Platform.

Power BI is a collection of software services, apps, and connectors that work together to turn unrelated sources of data into coherent, visually immersive, and interactive insights. Data may be input by reading directly from a database, webpage, PDF, or structured files such as spreadsheets, CSV, XML, JSON, XLSX, and SharePoint.

The parts of Power BI

Power BI consists of several elements that all work together, starting with these three basics:

• A Windows desktop application called Power BI Desktop.
• An online software as a service (SaaS) service called the Power BI service.
• Power BI Mobile apps for Windows, iOS, and Android devices.

These three elements—Power BI Desktop, the service, and the mobile apps—are designed to let users create, share, and consume business insights.

How Power BI Works: From Data to Insights

Power BI’s workflow can be summarized in three main steps: Connect, Transform, and Visualize.

• Connect: Power BI allows you to connect various data sources, including Excel files, databases, cloud services, and APIs.
  
• Transform: Once connected, you can clean and shape your data using Power Query Editor. This step involves removing errors, merging datasets, and creating calculated columns to prepare your data for analysis.
  
• Visualize: After transforming your data, you can start creating interactive visualizations using the built-in chart types, custom visuals, and filters.

Why Power BI is a Game-Changer for Businesses

1. Accessibility: One of Power BI’s greatest strengths is its accessibility. It’s designed for business users, not just IT professionals or data scientists. With an intuitive interface and a wealth of online resources, users can start creating reports and dashboards with minimal training.
   
2. Cost-Effective: Compared to other business intelligence tools, Power BI offers a cost-effective solution with its freemium model. Small businesses can start with Power BI Desktop, which is free, and scale up to Power BI Pro or Premium as their needs grow.
   
3. Real-Time Insights: In today’s fast-paced business environment, real-time data is crucial. Power BI allows users to connect to live data sources and build dashboards that update in real time. This ensures that decision-makers always have access to the most current information.
   
4. Scalability: Power BI is suitable for businesses of all sizes, from small startups to large enterprises. It can handle small datasets in Excel as well as massive data warehouses, providing a scalable solution that grows with the business.
   
5. Integration with Other Tools: Power BI’s ability to integrate with other Microsoft products, as well as third-party services, makes it a versatile tool in any business’s technology stack. Whether we are working in Excel, managing projects in Azure, or collaborating in Teams, Power BI can enhance our workflow.

Hands-On Exercise: Creating a Sales Dashboard

In this exercise, we’ll create a simple sales dashboard using sample data. Follow these steps to get started:

Step 1: Download and Install Power BI Desktop

1. Go to the Power BI website.

2. Download and install Power BI Desktop, which is free to use.

Step 2: Import Sample Data

For this exercise, we’ll be using a sample dataset, which you can download below,

1. Open Power BI Desktop.

2. Click on Home > Get Data > Excel.

3. Browse and select the sample Excel file.

4. In the Navigator window, select the worksheet that contains your data, and click Load.

Step 3: Clean and Transform the Data

Before visualizing the data, let’s make sure it’s clean and ready for analysis.

1. Click on Transform Data to open the Power Query Editor.

2. Here, you can remove any unwanted columns, rename columns, or change data types. For example, ensure the “Date” column is formatted as a Date type.

3. Once done, click Close & Apply.

Step 4: Create Visualizations

Now that your data is ready, it’s time to create some visualizations.

Bar Chart

1. Click on Bar Chart from the Visualizations pane.

2. Drag the “Product” field to the Y-axis and the “Sales” field to X-axis.

3. This will create a bar chart showing total sales by product.

Line Chart

1. Click on Line Chart from the Visualizations pane.

2. Drag the “Date” field to the X-Axis and the “Sales” field to Y-Axis.

3. This will create a line chart showing sales trends over time.

4. Clicking on “Expand all down one level in the hierarchy” option will display the chart in terms of month and year.

Map

1. Click on the Map visualization.

2. Drag the “Country” field to Location and the “Sales” to Size.

3. This will create a map showing sales distribution by country.

Step 5: Create a Dashboard

Let’s bring all these visuals together into a single dashboard.

1. Arrange your charts on the report canvas by dragging and resizing them.

2. You can add slicers to filter data interactively. For instance, add a slicer for the “Date” field to filter data by specific time periods.

3. Customize the visuals using formatting options like colours, labels, and titles.

Step 6: Save and share your report

Once your dashboard is complete:

1. Click on File > Save As to save your report.

2. To share it, publish it to the Power BI Service by clicking Publish on the Home ribbon, and then choose where to save it.

3. The published report can be seen in Power BI service https://app.powerbi.com/

Congratulations on completing the exercise! You’re one step closer to mastering Power BI.

­­By now, you’ve seen how easy it is to get started with Power BI. Through this hands-on exercise, you’ve learned how to connect to data, clean it up, and create interactive visualizations, all key steps in turning raw information into useful insights.

Power BI’s user-friendly interface and powerful features make it an ideal tool for anyone looking to dive into data analytics. With just a little practice, you’ll be able to create impactful reports and dashboards that can help drive smarter decisions in your business. Keep exploring Power BI, experiment with your own data, and start turning your insights into action.

Introduction

In the dynamic world of business, companies are always looking for innovative solutions to enhance competitiveness, drive down costs, and augment profits while embracing sustainability. Enter Artificial Intelligence (AI), a transformative tool that goes beyond mere automation, particularly with the advent of generative AI. This blog aims to explore the deeper layers of how companies can not only leverage AI to cut costs and boost profits but also contribute to building a sustainable future.

1. Automation

At its core, AI’s role in automation extends far beyond streamlining routine processes. Integrating AI into automation processes enables a more nuanced understanding of data, allowing for predictive analysis and proactive decision-making. This, in turn, minimizes downtimes and optimizes resource allocation. Moreover, AI-driven automation facilitates the identification of inefficiencies and bottlenecks that may go unnoticed in traditional systems, enabling companies to fine-tune their processes for maximum efficiency. In terms of cost reduction, AI excels in repetitive and rule-based tasks, reducing the need for manual labor and minimizing errors. Beyond the financial benefits, incorporating AI into automation aligns with sustainability goals by optimizing energy consumption, waste reduction, and overall resource management.

2. Predictive Analytics

AI’s real-time data processing capabilities empower companies with predictive analytics, offering a glimpse into the future of their operations. By analyzing historical data, AI forecasts market trends, customer behaviors, and potential risks. Consider a retail giant utilizing AI algorithms to predict customer preferences. This not only optimizes inventory management but also contributes to waste reduction and sustainability efforts.

By predicting future market trends, customer behavior, and operational needs, businesses can optimize their resource allocation, streamline operations, and minimize waste. This not only trims costs but also enhances profitability by aligning products and services with market demands. Moreover, predictive analytics enables companies to anticipate equipment failures, preventing costly downtime and contributing to a more sustainable operation. Harnessing the power of AI in predictive analytics is not just about crunching numbers; it’s about gaining insights that empower strategic decision-making, fostering a resilient and forward-thinking business model.

3. Personalization at Scale

Generative AI enables hyper-personalization by analyzing vast datasets to understand individual preferences, behaviors, and trends. Companies can utilize advanced algorithms to tailor products or services in real-time, offering a personalized experience that resonates with each customer. This not only fosters customer satisfaction but also drives increased sales and brand loyalty. On the cost front, AI streamlines operations through predictive analytics, optimizing supply chain management, and automating routine tasks. This not only reduces operational expenses but also enhances efficiency. In terms of sustainability, AI aids in resource optimization, minimizing waste and energy consumption. By understanding customer preferences at an intricate level, companies can produce and deliver exactly what is needed, mitigating excess production and waste.

4. Supply Chain Optimization

AI’s pivotal role in optimizing supply chains is revolutionizing sustainability efforts. Generative AI aids in demand forecasting, route optimization, and inventory management, minimizing waste and reducing the carbon footprint. Retail giants like Walmart have successfully implemented AI-powered supply chain solutions, resulting in substantial cost savings and environmental impact reduction.

AI can optimize various facets of the supply chain, from demand forecasting to inventory management. By analyzing historical data and real-time information, AI algorithms can make accurate predictions, preventing overstock or stockouts, thereby minimizing waste and maximizing efficiency. Additionally, AI-driven automation in logistics can streamline operations, cutting down on manual errors and reducing labor costs. Route optimization algorithms can optimize transportation, not only saving fuel and time but also curbing the carbon footprint. Predictive maintenance powered by AI ensures that equipment is serviced proactively, preventing costly breakdowns. Overall, the integration of AI into supply chain processes empowers companies to make data-driven decisions, fostering agility and resilience, ultimately translating into reduced costs, increased profits, and a more sustainable business model.

5. Predictive Maintenance

Generative AI’s impact extends to equipment maintenance, transforming the game by predicting machinery failures. Analyzing data from sensors and historical performance, AI algorithms forecast potential breakdowns, enabling proactive maintenance scheduling. This not only minimizes downtime but also significantly reduces overall maintenance costs, enhancing operational efficiency.

Picture this: instead of waiting for equipment to break down and incurring hefty repair costs, AI algorithms analyze historical data, sensor inputs, and various parameters to predict when machinery is likely to fail. This foresight enables businesses to schedule maintenance precisely when needed, minimizing downtime and maximizing productivity. This involves not just reacting to issues but proactively preventing them. By harnessing AI for predictive maintenance, companies can extend the lifespan of equipment, optimize resource allocation, and, ultimately, boost their bottom line. Moreover, reducing unplanned downtime inherently aligns with sustainability goals, as it cuts down on unnecessary resource consumption and waste associated with emergency repairs.

6. Fraud Detection

The ability of AI to detect patterns and anomalies proves invaluable in combatting fraud. Financial institutions, for instance, deploy generative AI to analyze transaction patterns in real-time, identifying potentially fraudulent activities. This not only safeguards profits but also bolsters the company’s reputation by ensuring a secure environment for customers.

AI systems can analyze vast datasets with unprecedented speed and accuracy, identifying intricate patterns and anomalies that might escape human detection. By deploying advanced machine learning algorithms, companies can create dynamic models that adapt to emerging fraud trends, ensuring a proactive approach rather than a reactive one. This not only minimizes financial losses but also reduces the need for resource-intensive manual reviews. Additionally, AI-driven fraud detection enhances customer trust by swiftly addressing security concerns. By curbing fraud, companies not only protect their bottom line but also contribute to sustainability by fostering a more secure and resilient business environment. It’s a win-win scenario where technology not only safeguards financial interests but aligns with the broader ethos of responsible and enduring business practices.

Conclusion

In conclusion, the integration of AI, especially generative AI, into business operations unveils many opportunities for companies seeking to reduce costs, increase profits, and champion sustainability. From the foundational efficiency of automation to the predictive prowess of analytics, and the personalized touch of generative AI, businesses can strategically utilize these tools for transformative outcomes. Supply chain optimization, predictive maintenance, and content creation further amplify the impact, showcasing the diverse applications of AI.

However, as organizations embark on this AI journey, ethical considerations and environmental consciousness must not be overlooked. Striking a balance between innovation and responsibility is paramount for sustained success. The future belongs to those companies that not only leverage AI for operational excellence but also actively contribute to creating a sustainable and equitable business landscape.

Introduction

Lately, there has been a viral buzz surrounding the term “generative AI.” It’s hard to scroll through social media without bumping into these mind-blowing, AI-generated hyper-realistic images and videos in various genres. These AI creations not only produce captivating visuals but also play a significant role in facilitating business growth, leaving us in awe.

While AI has been an integral part of our lives for quite some time, the current surge in creativity and complexity displayed in these generative creations makes it challenging when delving deeper into its workings.

If you’re an aspiring data analyst, machine learning engineer, or other professional who wishes to understand the basics of AI, this guide is for you. Let’s explore the different evolutions of artificial intelligence and the science behind it in simpler terms, and we’ll also delve into the top service providers of AI and how businesses leverage them in today’s landscape.

What is Artificial Intelligence?

Artificial Intelligence refers to the capability of machines to imitate human intelligence. This isn’t about robots replacing humans; rather, it’s the quest to make machines smart, enabling them to learn, reason, and solve problems autonomously.

AI empowers machines to acquire knowledge, adapt to changes, and independently make decisions. It’s like teaching a computer to think and act like a human.

Machine Learning

AI, or artificial intelligence, involves a crucial element known as machine learning (ML). In simpler terms, machine learning is akin to training computers to improve at tasks without providing detailed instructions. Machines utilize data to learn and enhance their performance without explicit programming. ML, a subset of AI, concentrates on creating algorithms for computers to learn from data. Instead of explicit programming, these systems use statistical techniques to continually improve their performance over time.

Prominent Applications of ML include:

Time Series Forecasting: ML techniques analyze historical time series data to project future values or trends, applicable in domains like sales forecasting, stock market prediction, energy demand forecasting, and weather forecasting.

Credit Scoring: ML models predict creditworthiness based on historical data, enabling lenders to evaluate credit risk and make well-informed decisions regarding loan approvals and interest rates.

Text Classification: ML models categorize text documents into predefined categories or sentiments, with applications such as spam filtering, sentiment analysis, topic classification, and content categorization.

Recommender Systems: ML algorithms are widely utilized in recommender systems to furnish personalized recommendations. These systems learn user preferences from historical data, suggesting relevant products, movies, music, or content.

While scaling a machine learning model to a larger dataset may compromise accuracy, another notable drawback is the manual determination of relevant features by humans, based on business knowledge and statistical analysis. Additionally, ML algorithms face challenges when handling intricate tasks involving high-dimensional data or complex patterns. These limitations spurred the development of Deep Learning (DL) as a distinct branch.

Deep Learning

Taking ML to the next level, Deep Learning (DL) involves artificial neural networks inspired by the human brain, mimicking how our brains work. Employing deep neural networks with multiple layers, DL grasps hierarchical data representations, automating the extraction of relevant features and eliminating the need for manual feature engineering. DL excels at handling complex tasks and large datasets efficiently, achieving remarkable success in areas like computer vision, natural language processing, and speech recognition, despite its complexity and challenges in interpretation.

Common Applications of Deep Learning:

• Autonomous Vehicles: DL is essential for self-driving cars, using deep neural networks for tasks like object detection, lane detection, and pedestrian tracking, allowing vehicles to understand and react to their surroundings.

• Facial Recognition: DL is used in training neural networks to detect and identify human faces, enabling applications such as biometric authentication, surveillance systems, and personalized user experiences.

• Precision Agriculture: Deep learning models analyze data from various sources like satellite imagery and sensors for crop management, disease detection, irrigation scheduling, and yield prediction, leading to more efficient and sustainable farming practices.

However, working with deep learning involves handling large datasets that require constant annotation, a process that can be time-consuming and expensive, particularly when done manually. Additionally, DL models lack interpretability, making it challenging to modify or understand their internal workings. Moreover, there are concerns about their robustness and security in real-world applications due to vulnerabilities exploited by adversarial attacks.

To address these challenges, Generative AI has emerged as a specific area within deep learning.

Generative AI

Now, let’s discuss Generative AI, the latest innovation in the field. Instead of just identifying patterns, generative AI goes a step further by actually producing new content. Unlike just recognizing patterns, generative AI creates things. It aims to produce content that closely resembles what humans might create

A notable example is Generative Adversarial Networks (GANs), which use advanced neural networks to make realistic content such as images, text, and music. Think of it as the creative aspect of AI. A prime example is deepfakes, where AI can generate hyper realistic videos by modifying and combining existing footage. It’s both impressive and a bit eerie.

Generative AI finds applications in various areas:

Image Generation: This involves the model learning from a large set of images and creating new, unique images based on its training data. The tool can generate imaginative images based on prompts like human intelligence.

• Video Synthesis: Generative models can generate new content by learning from existing videos. This includes tasks like video prediction, where the model creates future frames from a sequence of input frames, and video synthesis, which involves generating entirely new videos. Video synthesis is useful in entertainment, special effects, and video game development.

• Social Media Content Generation: Generative AI can automate content creation for social media platforms. By training models on extensive social media data, such as images and text, these models can produce engaging and personalized posts, captions, and visuals. The generated content is tailored to specific user preferences and current trends.

In a nutshell, AI is the big brain, Machine Learning is its learning process, Deep Learning is the intricate wiring, and Generative AI is the creative spark.

From spam filters to face recognition and deep fakes, these technologies are shaping our digital world. It’s not just about making things smart; it’s about making them smart in a way that feels almost, well, human.

Top Companies Leveraging AI in their Business:

As AI continues to advance and assert its influence in the business realm, an increasing number of companies are harnessing its capabilities to secure a competitive edge. Below are instances of businesses utilizing AI systems to optimize their operations:

Amazon: The renowned e-commerce retailer uses AI for diverse functions such as product recommendations, warehouse automation, and customer service. Amazon’s AI algorithms scrutinize customer data to furnish personalized product suggestions, while AI-powered robots in its warehouses enhance the efficiency of order fulfillment processes.

Netflix: This streaming service leverages AI to analyze user data and offer personalized content recommendations. By comprehending user preferences and viewing patterns, Netflix personalizes the viewing experience, ultimately boosting user engagement and satisfaction.

IBM: The multinational technology company utilizes its AI platform, Watson, across various sectors for tasks like data analysis, decision-making, and customer service. Watson adeptly analyzes extensive volumes of both structured and unstructured data, enabling businesses to obtain valuable insights and make more informed decisions.

Google: The prominent search engine giant integrates AI for search optimization, language translation, and advertising. Google’s AI algorithms possess the capability to comprehend and process natural language queries, deliver more precise search results, and furnish personalized advertising based on user data.

Conclusion

In conclusion, the rise of generative AI has undeniably captivated our imagination, showcasing its potential not only in creative endeavors but also as a driving force behind business growth.

As we witness the impressive applications of AI in companies like Amazon, Netflix, IBM, and Google, it becomes evident that AI’s transformative influence on various industries is profound.

Looking ahead, the question arises: What might follow generative AI? Could it be interactive AI? As businesses continue to embrace and leverage AI capabilities, the evolution of this technology holds the promise of more interactive and human-like experiences.

SAP Business Objects Data Services delivers a single enterprise-class solution for data integration, data quality, data profiling, and text data processing that allows you to integrate, transform, improve, and deliver trusted data to critical business processes. With SAP BusinessObjects Data Services, IT organizations can maximize operational efficiency with a single solution to improve data quality and gain access to heterogeneous sources and applications.

The important functions of SAP BODS are:

• Extraction Transformation and Loading (ETL): ETL can extract the data from any database or table and load the data into any other database or table.
• Data warehousing: Database is designed and developed in a particular format for data analysis and reporting. It can be implemented by using data from databases or data sources.
• Data Migration: Data migration is the process of moving data from one place to another place. It is a subset of ETL where the data is to be relocated from one software system or from one database to another database.
• Business Intelligence: It analyses the data of an organization effectively and helps in improving business performance.

Logging into the SAP BODS Designer:

You must have access to a local repository to log into the software. Typically, you create a repository during installation. However, you can create a repository at any time using the Repository Manager and configure access rights within the Central Management Server.

1. Enter your user credentials for the CMS.

2. Click Log on. The software attempts to connect to the CMS using the specified information. When you log in successfully, the list of local repositories that are available to you is displayed.
3. Select the repository you want to use.
4. Click OK to log in using the selected repository

BODS – Object Hierarchy

Designer window

The Designer user interface consists of a single application window and several embedded supporting windows.

1. Project area: Contains the current project (and the job(s) and other objects within it) available to you at a given time. In the software, all entities you create, modify, or work with are objects.
2. Workspace: The area of the application window in which you define, display, and modify objects.
3. Local object library: Provides access to local repository objects including built-in system objects, such as transforms, and the objects you build and save, such as jobs and data flows.
4. Tool palette: Buttons on the tool palette enable you to add new objects to the workspace.

Creating Datastores in DS Designer

To develop Data Migration work, you first need to create data stores for the source and the target system.

Step 1:

Click Create Data Stores.

A new window will open.

Step 2:

Enter the Datastore name, Datastore type, and database type as shown below. You can select a different database as the source system as shown in the screenshot below. And also, we need to mention the credentials for that particular database.

Step 3:

Click OK and the Datastore will be added to the Local object library list. If you expand Datastore, it does not show any table.

Data Migration Flow

Step 1:

Create a new project. Click the option, Create Project. Enter the Project Name and click Create. It will be added to the Project Area.

Step 2:

Right click on the Project name and create a new batch job/real time job.

Step 3:

Enter the name of the job and press Enter. You have to add Workflow and Data flow to this. Select a workflow and click the work area to add to the job. Enter the name of the workflow and double-click it to add to the Project area.

Step 4:

In a similar way, select the Data flow and bring it to the Project area. Enter the name of the data flow and double-click to add it under the new project.

Step 5:

Now drag the source table under datastore to the Work area. Now you can drag the target table with a similar data type to the work area or you can create a new template table.

To create a new template table right click the source table, Add New → Template Table. Or we can select it from the tool palette: Click the template table icon and drag it inside a data flow to place the template table in the workspace.

Step 6:

Drag the Query Transform to the workspace. Drag and connect them using the line from the source table to query transform and query transform to the target table. Click the Save All option at the top of the project menu.

Step 7:

Click on the Query transform and map the source Schema In columns that you want to include in the target table by dragging.

QueryTransform:

• Query Transform is similar to a SQL SELECT statement.
• It can perform the following operations-
  • Choose (filter) the data to extract from sources
  • Join data from multiple sources
  • Map columns from input to output schemas
  • Perform transformations and functions on the data
  • Add new columns, nested schemas, and function results to the output schema
  • Assign primary keys to output columns
  • Different functions can be performed using the query transform like LOOKUP, AGGREGATE, CONVERSIONS, etc.

Step 8:

Click the Save All option at the top of the project menu. Now you can schedule the job using Data Service Management Console or you can manually execute it by right clicking on the Job name and execute.

Once the job execution is complete, the data is transferred from the source to target databases based on the conditions that we have given in the query transform.

In conclusion, SAP Business Object Data Services (BODS) is a GUI tool which allows you to create and monitor jobs which take data from various types of sources and perform some complex transformation on the data as per the business requirement and then will load the data to a target which again can be of any type (i.e., SAP application, flat file, any database).

CloudIQ Attains New Microsoft Solutions Partner Designations

CloudIQ is proud to announce that we have attained three Microsoft Solutions Partner designations under the new Microsoft Cloud Partner Program – Azure Infrastructure, Data & AI, and Digital & App Innovation. The new partner program replaces Microsoft Silver and Gold competencies with new Solutions Partner Designations.

Microsoft Solutions Partner Designations

For each of the six Solutions Partner Designations – Infrastructure, Data & AI, Digital & App Innovation, Modern Work, Security, and Business Applications under the new Microsoft Cloud Partner Program (MCPP) partners must meet the requirements in three different categories which are Performance, Skilling, and Customer Success.

And we are happy to share that we have attained three of the six solution partner designations.

Microsoft Specializations

On top of the Solution Partner Designations, Microsoft also has advanced specialization programs that help demonstrate advanced technical expertise.

CloudIQ has earned advanced specialization in the Modernization of Web Applications to Azure and Kubernetes on Microsoft Azure.

Attaining the Microsoft Solution Partner Designations and Microsoft Advanced Specializations goes to show CloudIQ’s expertise and commitment to delivering best-in-class solutions for customers in any scenario and every industry.

We leverage our deep industry expertise to help businesses envision new products, create innovative business models, and deliver the next level of customer experiences by leveraging the cloud. From standalone cloud projects to enterprise-wide cloud architecture design you can rely on our cloud engineering expertise.

Get in touch with us to learn more.

Flutter is an open-source UI software development kit created by Google. It is used to develop cross platform applications for Android, iOS, Linux, macOS, Windows, and the web from a single codebase. Flutter apps are written in the Dart language.  Dart compiles to native machine code and hence it is optimized and has high performance.

Flutter is inspired by React Native, but with a few key differences: Flutter supports Hot Reload, which allows developers to make changes to production code without performing an app restart; Flutter uses the same rendering engine as Android, while React Native has historically used its own custom renderer. Flutter uses JavaScript, while React Native uses its own language – both have their strengths and weaknesses. Some may find Flutter easier to learn or be more familiar with, while others may prefer React Native.

Creating a new flutter app

After installing flutter on your machine, you can create a flutter project by using flutter create command.

We can also create the project using IDEs like Visual Studio Code or Android Studio.

The main.dart file in lib folder is where we build our app.

We can run the sample app by on an android emulator created using Android studio.  The sample app displays the number of times we have pressed the + symbol.

Widgets

Flutter has a unique architecture that makes it easy to develop cross-platform mobile apps. Its architecture is built around a widget tree. This means that all the widgets and components are arranged in a tree structure. In Flutter, you can create your own widgets and reuse them in any project.

Material widgets implements the Material design language for iOS, Android, and web.  Cupertino widgets implements the current iOS design language based on Apple’s Human Interface Guidelines.  We mostly use the Material widgets in our code.

Some common widgets are:

1. Scaffold: Implements the basic material design visual layout structure.  will occupy its entire window or device screen.

2. AppBar: AppBar is usually the topmost component of the app, it contains the toolbar and some other common action buttons.

All remaining widgets in a Scaffold other than AppBar are usually defined in the ‘body’ property of the Scaffold.

3. Text: Used to display formatted text in the app.

4. Column: A widget that displays its children in a vertical array.

5. Row: A widget that displays it’s children in the horizontal direction.

The tree of widgets displayed in the sample app is given below:

Stateless and Stateful widgets

The widgets whose state cannot be altered once they are built are called stateless widgets.  Below is the basic structure of a stateless widget. Stateless widget overrides the build () method and returns a widget.

import 'package:flutter/material.dart';
 
class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Container();
  }
}

The widgets whose state can be altered once they are built are called stateful Widgets.  Below is the basic structure of a stateful widget. Stateful widget overrides the createState() and returns a State. It is used when the UI can change dynamically.

import 'package:flutter/material.dart';
 
class MyApp extends StatefulWidget {
 
  @override
  // ignore: library_private_types_in_public_api
  _MyAppState createState() => _MyAppState();
}
 
class _MyAppState extends State<MyApp> {
  @override
  Widget build(BuildContext context) {
    return Container();
  }
}

Whenever we change the values of properties denoting the state, we must wrap it by a call to setState function to inform flutter to rebuild the widget and display it.

setState(() {
      _counter++;
    });

Hot Reload

Flutter allows hot reloading during development.  Hot reloading allows us to keep the app running and to inject new versions of the files that we edited at runtime.  This way, we don’t lose any of our state which is especially useful if we are making the UI changes.  For example, in our sample code if we change the primarySwatch to Colors.red, the app color changes from blue to red, but the counter still shows 1 and doesn’t get reset to 0.

In conclusion, Flutter is a framework that is built on the Dart programming language and can be used to create native apps for Android and iOS. Flutter uses a widget-based architecture, and so you’ll be able to create apps that look and feel like the ones that you’ve seen on the Apple and Android stores.

When systems evolve the need to migrate database does arise based on the data we are dealing with. In one of our projects, we had a scenario where non-transactional data had to be accessed frequently with low latency across regions. Azure Cosmos DB is Microsoft’s fast NoSQL database and the first globally distributed database service in the market today to offer comprehensive service level agreements encompassing throughput, latency, availability, and consistency.

So, the choice was clear for us to move the data from PostgreSQL to CosmosDB. In PostgreSQL the data format is flat (i.e., in the form of tables and columns), while CosmosDB supports flexible schemas and hierarchical data, and thus it is well suited for storing catalog data. Provides. JSON format supported by Cosmos DB is an effective format that is very lightweight.

For the data migration from PostgreSQL to CosmosDB we chose Azure Data Factory. Azure Data Factory helps you integrate, perform transformations, and visualize all your data with ease. Azure Data Factory is easy to use, cost-effective, a fully serverless cloud service that accelerates data transformation with code-free data flows.

Data Transformation using Azure Data Factory

Azure Data Factory is an orchestration tool that is used to transform the data from one source to another. It can process and transform the raw data into predictions and insights. It allows you to perform data transformation activities via pipelines. Data flows are created in debug mode to validate the logic of the transformation. The data flow activity is added to the pipeline to execute and test the data flow. And “trigger now” is employed to test the data flow that is in the pipeline.

Several options are available for converting JSON data to flat data, however when it comes to converting flat data into JSON, it is still a challenge. With JSON, inserting null value within the data flow is quite tedious and a unique pipeline must be created to handle null values.

The Azure Data factory has three main options:

• Author
• Monitor
• Manage

The Author option provides the main environment for development. Using this option, we can design and manage Azure Data Factory resources such as the pipelines and dataflows.

The Monitor option allows you to monitor the pipelines; trigger runs; sessions; the time taken for execution and running the pipelines; check if the execution of the pipeline was a success or failure and set up alerts.

The Manage option allows you to manage the connections, link service, source control, triggers, parameters, and security.

Now let’s look at how to migrate from PostgreSQL to CosmosDB and transform the data.

Pipeline and activities creation

The Author option provides the environment for the development of pipelines and data flows. So, the first step is to create a pipeline that contains the data flow activity. The pipeline is a logical grouping of activities that perform a task. By clicking on orchestrate on the home page, we create and name the pipelines.

The activities pane allows you to perform various activities such as move, transform, add data flow (we can use existing data flows or even create new data flows), Azure data explorer, Azure function, Batch Service, Databricks, Data Lake Analytics, etc. Once the data flow is created, we can provide the transformation logic in the data flow canvas. The dataflows distribute the processing of data over different nodes in a Spark cluster to perform the operations parallelly. We need to create a mapping data flow to perform the transformation as well.

We can choose the format of the data as per the requirement. We can also select the dataset, which is simply the view of data or the references of the data that you want to employ in your activity.

In our data migration scenario, say for an activity to copy data from the source (PostgreSQL data directory), the data is taken from the source and put in the blob storage. The basic details are stored in the first copy activity, the patient color details are stored in the second, microchip details are stored in the third, breed details in the fourth and existing patient details are stored in the fifth copy activity respectively. The data flow performs transformations to the data such as join, conditional join, aggregate, pivot, flatten, union, split, etc. And finally, triggers are scheduled for doing transformation in the pipelines.

Linked Service and Integration Runtime

When creating a dataset, first we must create a linked service to link the data store to Data Factory via the management hub. Linked services are much like connection strings and defines the connection to the data source.

In our data migration scenario, we have created the following linked services for Azure Managed Instance, Azure Blob storage, CosmosDB (one for dev and one for UI), and PostgreSQL. To create the linked service, we must provide the server’s name, port, database name, username, password, etc. These are the input for dataflow to connect to the specific database. In this migration, the PostgreSQL data is migrated from a different VM environment into the Azure environment.

Here, the base directory is called containers, inside the containers we have several directories which further contain storage files.

The integration runtime via the management hub is the compute infrastructure for providing data integration capabilities such as data flow, data movement, activity dispatch, and SSIS package execution across different network environments. It provides the linkage between the activity and linked services. The default option is public for the Auto-resolve integration runtime. We can also create private customized ones based on the requirement for dataflow execution.

We hope this walkthrough of migrating data from PostgreSQL to Azure CosmosDB using Azure Data Factory was helpful. If you have queries or want us to help you with your data migration projects, feel free to reach out to us.

Angular is a component-based application design framework for building scalable, reactive, and efficient single-page apps. It provides a wide range of tools and well-integrated libraries to build, develop and test your applications. The angular applications are written by composing HTML templates and components are created to manage these HTML templates. The logic of the applications is given in the services, these services and components are wrapped into the modules.

Angular Evolution

Angular framework was first introduced a decade ago in 2010 under the name Angular JS. Through the course of years, the framework has developed with various updates. AngularJS 1.x is seen as a JavaScript-based framework that creates rich web and mobile applications. The Model-View-Controller (MVC) and its variations Model-View-Presenter (MVP) and Model-View-View-Model (MVVM) have been considered as an Angular JS architecture. Even though Angular JS is a proven architectural pattern and is the standard solution for an application with a view, there are certain disadvantages.

• Memory leakage.
• Internet Explorer 8.0 doesn’t support Angular JS.
• AngularJS totally depends on JavaScript.

Angular 2+ was typescript-based free open-source technology used to develop web applications and mobile applications. It is a component-based framework and has a collection of well-integrated libraries. Angular 2+ is very simple and pretty straightforward to use. Angular 2+ is typescript, which allows you to validate the code with ease and show the error at the time of typing. Implementing forms and validation are much simpler and more effective with angular 2+. With recent advancements, Agular’s responsive design has shifted towards a mobile-first approach.

Angular 3 was skipped because the framework was developed in a MonoRepo, which is a single repo, and its router package was already in the third version. To avoid confusion in terms of dependency, Angular third version was skipped.  Angular 4 was released in 2017 which was compatible with both typescript 2.1 and 2.2.  This version improved the speed and performance by a good extent. Angular 5 was released around the end of 2017 and it was comprised of many new features and improvements. It provided an optimizer that removed unnecessary code from the applications.           It also provided an improved compiler and improvements in angular universal for code allocation, and most important of all it supported typescript 2.4. Angular 6 was released in early 2018 which introduced Angular elements and Angular Rendering Engine.  Angular 7 was released later that year with major performance improvements. It also provided a drag and drop module, angular material, and component Dev kit. Angular 8 was released in 2019 with its dependencies updated, improved web worker bundling, and provided a new lazy loading syntax and also includes angular firebase.

Angular 9 was introduced in 2020, it provided new features and updates such as more consistent ng, updated and improved API extractor, dependency injection update,  better speed and performance, AOT build ensures a faster and better performing compiler, supports typescript 3.7.  Angular 10 was released in June 2020 with new updates and features such as a new data range picker, an updated compiler, Optional stricter settings to catch bugs ahead, performance improvements, supports typescript 3.9.

Angular 11 was released in November 2020, major improvements are router performance, automatic inlining of fonts, updated Hot Module Replacement(HMR), provides faster builds and improved performance, supports Typescript 4 and Webpack 5, component test harness with parallel functions and improved performance. Angular 12 was released on May 2021, with major improvements in styling, supports Typescript 4.2 and Webpack 5.3.7,  nullish coalescing for writing better and cleaner code in typescript classes, improved ng, includes protractor, new dev tools, Migrating from legacy i18n message IDs. Angular 13 is also available with impactful changes towards optimizations. Angular 13 no longer has the view engine, with reduced dependency on ngcc, we can hope for faster and improved compilation.  Angular 13 supports typescript 4.4 and provides an improved and modernized Angular Package Format,  improved Angular CLI and it no longer supports IE11.

Why Angular?

With the evolution of Angular over the years, it has become one of the most recommended for businesses and enterprises for various reasons. Angular is used to develop single-page client applications using HTML and TypeScript. Angular offers two-way data binding and it shares the data between Model as well as View. Hence, when data is modified or changed, components get updated automatically in real-time. Code reusability in an angular application is high.

Angular makes a great recommendation to businesses as it provides a framework that can work well with back-end languages and also very well combines the business logic and UI. Angular provides an effective cross-platform development framework that makes the development process easier with reduced cost. Though initially, it is a little complex to learn, it is worth it as it provides high-quality applications.   Angular provides Typescript that facilitates the developers to write clean and neat code, which makes the fixing of the bugs that much easier. The framework is structured around component-based development, which aids in a steady development process with consistent and high reusability of components. This further facilitates in providing better maintainability and productivity.

The evolution of angular has provided various new features and improvements that have significantly increased the speed and optimized the performance. The older versions had larger bundles size which hindered the fast loading of the applications, but with recent improvements such as lazy load modules and Ivy Renderer, we can create lightweight web applications that are faster and better. To overcome productivity issues and to provide a faster development process, features such as dependency injection and angular services are also provided. Angular keeps ever-evolving based on requests from google and the angular community making it one of the ideal frameworks for businesses and enterprises.

 Angular Architecture and it’s core components

The angular architecture contains the following core components

• Module
• Meta Data
• Directives
• Pipe
• Service
• Decorators

Module

Module in Angular refers to a place where you group the components, directives, pipes, and services, which are related to the application. Every Angular app has a root module that has the bootstrap mechanism to launch the app.

Meta Data

Metadata is used to decorate a class so that it can configure the expected behavior of the class. Metadata is attached to TypeScript using the decorator.

Directives

Directives are classes that are used to change the behavior or view of DOM elements in Angular applications. 

Three types of Angular directives are as follows:

1. Components – directives with a template.
2. Attribute directives – directives that change the appearance or behaviour of an element, component, or another directive.
3. Structural directives – directives that change the DOM layout by adding and removing DOM elements.

Pipe

The pipe is asimple way to transform values in an Angular template. Some of the built-in pipes in angular are CurrencyPipe, DatePipe, JsonPipe, LowerCasePipe, UpperCasePipe, PercentPipe, SlicePipe.

Service

 Services are used to access and share the methods and properties with the other components in the entire project. It reduces the function and properties repetitions in the entire project. HTTP requests and responses will be handled in the services.

There are two types of services in angular.

• Built-in services – There are approximately 30 built-in services in angular.
• Custom services – In angular, if the user wants to create their own service, they can opt for custom services.

Decorators

Angular decorators are used to storing metadata. It is about a class, method, or property. There are four types of decorators in Angular:

1. Class Decorators
2. Property Decorators
3. Method Decorators
4. Parameter Decorators

1. Class Decorators

Class Decorators are top-level decorators. It defines the purpose of the class.

2. Property Decorators

Property decorators are used to specific properties within the class. @Input() is the example for the property decorator.

3. Method Decorators

A Method Decorator is used to specify the methods within your class with functionality. @HostListener() is an example of a method decorator.

4. Parameter Decorators

Parameter decorators are used to decorating parameters. It is used in the class constructors. @Inject() is the example for the parameter decorator

Testing 

Angular uses the Jasmine testing framework. It provides multiple functionalities to test. Karma is the task-runner, uses a configuration file to set the start-up, reporters, and testing framework.

Limitations of Angular

Angular though it is one of the popular modern-day frameworks that is employed, it also has a few limitations:

• Steep learning curve

Though Angular is a great framework, it’s sometimes is quite difficult even for people with experience in HTML, JS, CSS, and for people who are not much used to n-tier architecture. They can find it hard to learn some of the concepts as it has its own set of rules, which might be difficult and uncomfortable for novice learners.

• Limited SEO options

Though Angular is a great and powerful platform to build single-page applications, limited SEO options and poor accessibility to search engine crawlers are some of the major drawbacks. This makes it difficult to place the website correctly in the list provided by the search engines.

• Complex and Verbose

Angular has such a complex framework of modules and complex capabilities for integration and customization. Though angular provides an array of online tutorials and documentation, it’s quite uncomfortable to learn in the beginning. A slow and steady pace is recommended to learn the platform and language.

• Complex directives

Angular has three directives -attribute directive, structural directive, and component directive. The three directives have their own limitations and it’s quite complex for beginners to understand when to use what.

Angular Prerequisites

Angular requires the following prerequisites – NodeJS, Angular CLI, and Text Editor

Supported by Google

Google is offering a Long-Term Support (LTS) for Angular to scale up to enterprise Angular applications development. Netflix, Gmail, YouTube TV, Upwork, and other organizations are also using the Angular framework for application development.

Open-Replay

Open Replay is an open-source tool, developers can integrate this with the angular applications. When the user uses the application, open-replay will track it. So the developers can easily come to know how much the application is user-friendly. It has the network tracker, redux, NgRx action tracker, and profile tracker plugins. It also captures the performance of the application.

With this article we wanted to give you an introduction to Angular, the evolution of angular over the decade, the need for angular in today’s business world, and its core architecture. We covered the core components of the architecture and also the limitations of the framework. Bottom line is that Angular has become one of the best frameworks for developing single page apps.

Azure App Service is a Platform as a Service (PaaS) that is used to build, deploy, and scale enterprise-grade applications such as web apps, mobile apps, logic apps, API apps, and function apps. It supports multiple programming languages and frameworks such as NET, .NET Core, Java, Ruby, Node.js, PHP, Python.

From a developers perspective Azure App Service provides a great platform to develop, deploy and scale applications.  However, when it comes to production environments Infrastructure as code (IaC) comes in handy. Terraform is an open-source IaC tool with a consistent CLI that lets you write infrastructure as code using declarative configuration files and also, manage, plan and apply changes to infrastructure versions to reach the required configuration state.

Terraform is a good choice as it reduces manual human error by codifying the application infrastructure. Terraform manages infrastructure across more than 300 public clouds and it provides a reusable, cost-effective, and consistent environment that solves dependencies and version controls. In this article, we will take you through the process of deploying a web app in Azure App Service using Terraform.

To deploy the web app in Azure App Service using Terraform, here are the steps we need to follow:

• Create the Resource Group
• Create App Service plan and deploy web app

Create the Resource Group:

The first step is to create a resource group using the following terraform code. Any resource that is created must be created within the resource group

terraform {    
  required_providers {    
    azurerm = {    
      source = "hashicorp/azurerm"    
    }    
  }    
} 
   
provider "azurerm" {    
  features {}    
}

resource "azurerm_resource_group" "resource_group" {
  name     = "app-service-rg"
  location = "East US"
}

Create App Service plan and deploy web app

The App Service plan defines the capacity and resources to be shared among one or more app services that are assigned to that plan. Azure WebApp must be associated with an App Service Plan as it specifies the computing resources that are required for the web app to function. The following code creates an app service plan.

resource "azurerm_app_service_plan" "app_service_plan" {
  name                = "example-appserviceplan"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  sku {
    tier = "Standard"
    size = "S1"
  }
}

And add code for creating app service. Finally, the terraform file looks like the below

terraform {    
  required_providers {    
    azurerm = {    
      source = "hashicorp/azurerm"    
    }    
  }    
} 
   
provider "azurerm" {    
  features {}    
}

resource "azurerm_resource_group" "resource_group" {
  name     = "app-service-rg"
  location = "East US"
}

resource "azurerm_app_service_plan" "app_service_plan" {
  name                = "myappservice-plan"
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name

  sku {
    tier = "Standard"
    size = "S1"
  }
}

resource "azurerm_app_service" "app_service" {
  name                = "mywebapp-453627 "
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name
  app_service_plan_id = azurerm_app_service_plan.app_service_plan.id

  #(Optional)
  site_config {
dotnet_framework_version = "v4.0"
    scm_type                 = "LocalGit"
  }
  
  #(Optional)
  app_settings = {
    "SOME_KEY" = "some-value"
  }

}

Now, we should run the following command to initiate terraform.

Command: terraform init

To create an execution plan, we should run the terraform plan command

Command: terraform plan -out appservice.tfplan

To apply the plan, run the following command

Command: terraform apply ” appservice.tfplan “

We can verify the app service created in the specified app service plan and resource group by checking in the Azure portal

Hope you found this article useful. Stay tuned for more articles coming up on Azure App Service and Terraform.

With an average cost of a data breach at $3.86 million last year, it’s wise to employ a good backup system. More than 80% of the fortune 500 companies use Microsoft Azure for running their businesses effortlessly as they are simple, ever-evolving, secure and cost effective. So, in this article let’s explore how to backup and restore Azure Managed Disks using Azure Backup Vault.

Azure Backup services allow you to back up your data and recover it from the Microsoft Azure cloud. They backup and store the data in the backup vaults. These backup vaults make certain that the backups are successful by monitoring and tracking the storage containers, they optimize the resources by automating maintenance tasks and they also provide better security and access control to store and recover data.

Data sources that are supported by Azure Backup include

• Azure Database for PostgreSQL servers,
• Azure Blobs, and
• Azure Disks.

Prerequisites for performing disk backup and restore operations

Backup Vault’s managed identity needs the below roles to be assigned to it for performing disk backup and restore operations:

• Disk Backup Reader role on the Source disk that needs to be backed up.
• Disk Snapshot Contributor role on the Resource group where backups are created and managed by Azure Backup.
• Disk Restore Operator role on the Resource group where the disk will be restored by the Azure Backup.

To assign Azure roles, the user must have Microsoft.Authorization/roleAssignments/write permissions, such as User Access Administrator or Owner.

Steps to backup managed disks:

Create a Backup vault

1. Go to Backup center service in Azure portal. Backup center enables enterprises to govern, monitor, operate, and analyze backups at scale. Jobs performed in last 24 hours are displayed in the Overview tab. Operations such as Scheduled backup, On-demand backup and Restore are listed along with the status of each operation (Failed, In progress or Completed)

2. Select Vault from the Overview tab

3. In Start: Create Vault page, select Backup vault and then Continue

4. In Basics tab,

• Under PROJECT DETAILS, select the Subscription and Resource group of the vault to be created.
• Under INSTANCE DETAILS, type in the Backup vault name.
• Select the region of the backup vault and backup storage redundancy
• Select Review and create

5. Select Create. The Backup vault will be created.

Create a backup policy

1. Select Policy from Backup center’s Overview tab

2. In Start: Create Policy page, select Datasource type as Azure Disks and Vault type is prepopulated as Backup vault. Then, select Continue.

3. In the Basics tab, type in the policy name to be created. Select Datasource type as Azure Disk and select Vault as the Backup vault that was just created. Click on Next: Schedule and Retention to go to next tab.

4. In the Schedule and retention tab,

• Under Backup schedule, select the backup schedule frequency and specify the time when backup must happen.
• Specify the number of days backup should be retained under Retention settings.

5. After validation, in Review and Create tab, select Create. The Backup policy is created.

Configure backup of an Azure Disk

To backup an azure disk,

1. Assign Disk Backup Reader role on the Source disk that needs to be backed up to the Backup vault’s managed identity.
2. Assign Disk Snapshot Contributor role on the Snapshot Resource group to the Backup vault’s managed identity.

a)     Steps to Assign Disk Backup Reader role on the Source Disk

1. Go to the source disk that we need to configure backup for.
2. Select access control (IAM) and Add role assignment

3. In Role tab, search for the role Disk Backup Reader and select it

4. In Members tab, select assign access to Managed identity and select members as the Backup vault.

5. Select Review+ assign. Assignment of Disk Backup Reader role to the Backup vault is done.

b)     Steps to Assign Disk Snapshot Contributor role on the Snapshot Resource group

1. Go to the target Snapshot resource group.
2. Select access control (IAM) and Add role assignment

3. In Role tab, search for the role Disk Snapshot Contributor and select it

4. In Members tab, select assign access to Managed identity and select members as the Backup vault.

5. Select Review+ assign. Assignment of Disk Snapshot Contributor role to the Backup vault is done.

Steps to Backup an Azure Disk

1. In Backup center, Select Backup from the Overview tab

2. In Start: Configure Backup, select Datasource type as Azure Disks and Vault type is prepopulated as Backup vault.

3. In Basics tab, select Datasource type as Azure Disks,select the Vault created and then Next.

4. In Backup policy tab, Select the backup policy.

5. In Datasources tab:

a) Click on Add/Edit and select the disks to backup.

b) Click Select after selection of disks.

c) Select Snapshot Resource Group, the resource groupwhere snapshots of disks are stored. Once the disk backup is configured, the Snapshot Resource Group that’s assigned to a backup instance cannot be changed.

d) Select Validate. Click Next.

6. In Review and Configure tab, Select Configure Backup. The configuration of backup for the disk is done.

For the validation to be successful, We must assign Disk Backup Reader role on the Source disk that needs to be backed up to the Backup vault’s managed identity and Disk Snapshot Contributor role on the Snapshot Resource group to the Backup vault’s managed identity.

On demand backup of an Azure Disk

1. In the Backup Vault, go to Backup Instances and select the disk to perform on demand backup

2. Select Backup Now.

3. In Backup vault, go to Backup jobs to view the status of the backup.

Restore an Azure Disk from backup

To restore an azure disk, we need to assign Disk Restore Operator role to Backup vaults managed identity on the Resource group where the disk will be restored by the Azure Backup.

Steps to Assign Disk Restore Operator role on the Target Resource group

1. Go to the target resource group.
2. Select access control (IAM) and Add role assignment

3. In Role tab, search for the role Disk Restore Operatorand select it

4. In Members tab, select assign access to Managed identity and select members as the Backup vault.

5. Select Review+ assign. Assignment of Disk Restore Operatorrole to the Backup vault is done.

Steps to Restore an Azure Disk

1. Go to Backup center -> select Backup vault -> select Restore

2. In Basics tab, Select the Backup instance as the disk that needs to be restored and then Next.

3. In Select Restore Point tab, select the required or latest restore point.

4. In Restore parameters, select Target subscription, and Target resource group. Type in the Restored disk name and select Next:Review and restore.

5. After validation, click on Restore. The restore operation is started.

Note: If validation is unsuccessful, follow the steps to Assign Disk Restore Operator role on the Target Resource group.

6. Restore operation is now completed.

While adopting DevOps practices automates and optimizes processes through technology, it all starts with the culture inside the organization—and the people who play a part in it.

Check out this infographic to learn how DevOps unifies people, process, and technology to bring better products to customers faster. Then imagine how the power of GitHub and Azure can benefit your DevOps team.

Together, Microsoft GitHub and Azure DevOps provides an end-to-end experience for development teams to easily collaborate while building and releasing code to Azure, on-premises or any cloud. Contact us today to learn more.

This infographic offers an in-depth look at how Microsoft business analytics and AI is intelligent, trusted, and flexible. This service produces faster, more accurate insights and predictions. It also offers the most secure, compliant, and scalable system. Finally, it works with what you have.

Would you like to leverage Microsoft Business Analytics and AI for faster, more accurate insights and predictions? At CloudIQ Technologies, we have knowledgeable and professional team ready to help you. Contact us today to learn more.

Seattle – [23 Jun 2021] – CloudIQ Technologies Inc today announced it has earned the Kubernetes on Microsoft Azure advanced specialization, a validation of a solution partner’s deep knowledge, extensive experience and proven expertise in deploying and managing production workloads in the cloud using containers and managing hosted Kubernetes environments in Microsoft Azure.

Only partners that meet stringent criteria around customer success and staff skilling, as well as pass a third-party audit of their container-based workload deployment and management practices, are able to earn the Kubernetes on Azure advanced specialization.

With over 75% of global organizations expected to run containerized applications in production by 2022, many are looking for a partner with advanced skills to migrate their existing containerized workloads to the cloud, or assist them in developing cloud-native applications using container technologies, DevOps patterns, and a microservices approach.

“With our deep expertise in cloud-native architecture design, we help clients build and run scalable applications with improved security, faster release cycles, easier management, and lower costs”, said Mr. Prem Kandalu, CEO. “As a partner who has earned the Kubernetes on Microsoft Azure advanced specialization, CloudIQ will pass on the benefits of our continued collaboration with Microsoft to our clients.”

Rodney Clark, Corporate Vice President, Global Partner Solutions, Channel Sales and Channel Chief at Microsoft added, “The Kubernetes on Microsoft Azure advanced specialization highlights the partners who can be viewed as most capable when it comes to deploying and managing containerized applications in Azure. CloudIQ Technologies clearly demonstrated that they have both the skills and the experience to deliver best-in-class cloud-native capabilities to customers with Azure.

About CloudIQ Technologies

CloudIQ is a leading cloud consulting and solutions firm that helps businesses envision new products, create innovative business models, and deliver the next level of customer experiences by leveraging the cloud. From standalone cloud projects to enterprise-wide cloud architecture design you can rely on our cloud engineering expertise.

As a Microsoft Gold Partner (Cloud Platform), earner of the Modernization of Web Applications on Microsoft Azure and Kubernetes on Microsoft Azure advanced specializations, Kubernetes Certified Service Provider (KCSP) and Kubernetes Training Partner (KTP), we serve as trusted advisors to Fortune 500 organizations and leverage our deep industry expertise in building cloud-native solutions to help our clients realize the cost, scale and security benefits of the cloud.

Without artificial intelligence (AI), organizing and extracting insights from vast amounts of enterprise data would be a nearly impossible task. Choosing the right AI capabilities is essential to successful initiatives. This infographic presents the four guiding principles behind Microsoft #Azure #AI and why it remains the top choice for today’s leading enterprises.

Would you like to leverage Azure AI for your business,? At CloudIQ Technologies, we have knowledgeable and professional team ready to help you transform massive amounts of raw information into meaningful insights for your business. Contact us today to learn more.

Have you been looking for a fully-managed, secure platform for your web apps? Azure App Service is built to help you build, deploy, and scale your web apps and APIs on your terms. Work with .NET, .NET Core, Node.js, Java, Python or php, in containers or running on Windows or Linux. Check out this infographic and contact CloudIQ Technologies to learn more.

Would you like to modernize your apps using Azure App Service? At CloudIQ Technologies, we have knowledgeable and professional team ready to help you. Contact us today to learn more.

Azure SQL Databases are intelligent and always up to date. It is the only cloud with evergreen SQL which never needs to be patched or updated. This infographic presents the benefits of Azure SQL Database and Azure Advance Threat Protection.

Would you like to migrate SQL Server databases to the Azure cloud,? At CloudIQ Technologies, we have knowledgeable and professional team ready to help address any of your IT infrastructure upgrade needs. Contact us today to learn more.

Moving Windows Server and SQL workloads to Azure provides flexible, scalable, and highly available cloud infrastructure. It also supports rapid innovation and digital transformation, freeing you to focus on your mission. This infographic presents the benefits of running Windows Server and SQL Server on Azure. 

Would you like to move your Windows Server and SQL workloads to Azure? At CloudIQ Technologies, we have knowledgeable and professional team ready to help address any of your IT infrastructure upgrade needs. Contact us today to learn more.

There are four different ways of accessing Azure Data Lake Storage Gen2 in Databricks. However, using the ADLS Gen2 storage account access key directly is the most straightforward option. Before we dive into the actual steps, here is a quick overview of the entire process

• Understand the features of Azure Data Lake Storage (ADLS)
• Create ADLS Gen 2 using Azure Portal
• Use Microsoft Azure Storage Explorer
• Create Databricks Workspace
• Integrate ADLS with Databricks
• Load Data into a Spark DataFrame from the Data Lake
• Create a Table on Top of the Data in the Data Lake

Microsoft Azure Data Lake Storage (ADLS) is a fully managed, elastic, scalable, and secure file system that supports HDFS semantics and works with the Apache Hadoop ecosystem.  It is built for running large-scale analytics systems that require large computing capacity to process and analyze large amounts of data

Features:

Limitless storage

ADLS is suitable for storing all types of data coming from different sources like devices, applications, and much more. It also allows users to store relational and non-relational data. Additionally, it doesn’t require a schema to be defined before data is loaded into the store. ADLS can store virtually any size of data, and any number of files. Each ADLS file is sliced into blocks and these blocks are distributed across multiple data nodes. There is no limitation on the number of blocks and data nodes.

Auditing

ADLS creates audit logs for all operations performed in it.

Access Control

ADLS provides access control through the support of access control lists (ACL) on files and folders stored in its infrastructure. It also manages authentication through the integration of AAD based on OAuth tokens from supported identity providers.

Create ADLS Gen2 using Portal:

1. Login into the portal.
2. Search for “Storage Account”
3. Click “Add”

4. Choose Subscription and Resource Group.

5. Give storage account name, location, kind, and replication.

6. In the Advanced Tab, set Hierarchical namespace to Enabled

7. Click “Review+Create”

Microsoft Azure Storage Explorer

Microsoft Azure Storage Explorer is a standalone app that makes it easy to work with Azure Storage data on Windows, macOS, and Linux.  Microsoft has also provided this functionality within the Azure portal which is currently in preview mode.1.

1. Navigate back to your data lake resource in Azure and click ‘Storage Explorer (preview)’.

2. Right-click on ‘CONTAINERS’ and click ‘Create file system’. This will be the root path for our data lake.

3. Name the file system and click ‘OK’.

4. Now, click on the file system you just created and click ‘New Folder’. This is how we will create our base data lake zones. Create folders.

5. To upload data to the data lake, you will need to install Azure Data Lake explorer using the following link.

6. Once you install the program, click ‘Add an account’ in the top left-hand corner, log in with your Azure credentials, keep your subscriptions selected, and click ‘Apply’.

7. Navigate down the tree in the explorer panel on the left-hand side until you get to the file system you created, double click on it. Then navigate into the folder. There you can upload/ download files from your local system.

8. Click “Upload” > “Upload Files”. You can get sample data set from here.

Sample Folder structure:

Create Databricks Workspace

1. On the Azure home screen, click ‘Create a Resource’

2. In the ‘Search the Marketplace’ search bar, type ‘Databricks’ and you should see ‘Azure Databricks’ pop up as an option. Click that option.

3. Click ‘Create’ to begin creating your workspace.

4. Use the same resource group you created or selected earlier. Then, enter a workspace name.

5. Select ‘Review and Create’.

6. Once the deployment is complete, click ‘Go to resource’ and then click ‘Launch Workspace’ to get into the Databricks workspace.

Integrate ADLS with Databricks:

There are four ways of accessing Azure Data Lake Storage Gen2 in Databricks:

1. Mount an Azure Data Lake Storage Gen2 filesystem to DBFS using a service principal and OAuth 2.0.
2. Use a service principal directly.
3. Use the Azure Data Lake Storage Gen2 storage account access key directly.
4. Pass your Azure Active Directory credentials, also known as a credential passthrough.

Let’s use option 3.

1. This option is the most straightforward and requires you to run the command, setting the data lake context at the start of every notebook session. Databricks Secrets are used when setting all these configurations

2. To set the data lake context, create a new Python notebook, and paste the following code into the first cell:

spark.conf.set(
"fs.azure.account.key.<storage-account-name>.dfs.core.windows.net",
""
)

3. Replace ‘<storage-account-name>’ with your storage account name.

4. In between the double quotes on the third line, we will be pasting in an access key for the storage account that we grab from Azure

5. Navigate to your storage account in the Azure Portal and click on ‘Access keys’ under ‘Settings’.

6. Click the copy button, and paste the key1 Key in between the double quotes in your cell

7. Attach your notebook to the running cluster and execute the cell. If it worked, you should see the following:

8. If your cluster is shut down, or if you detach the notebook from a cluster, you will have to re-run this cell to access the data.

9. Copy the below command in a new cell, filling in your relevant details, and you should see a list containing the file you updated.

dbutils.fs.ls("abfss://<file-system-name>@<storage-account-
name>.dfs.core.windows.net/<directory-name>")

Load Data into a Spark DataFrame from the Data Lake

Towards the end of the Error! Reference source not found. section, we uploaded a sample CSV file into ADLS.  We will now see how we can read this CSV file from Spark.

We can get the file location from the dbutils.fs.ls command we ran earlier – see the full path as the output.

Run the command given below:

#set the data lake file location:
file_location = "abfss://[email protected]/raw/covid
19/johns-hopkins-covid-19-daily-dashboard-cases-by-states.csv"
 
#read in the data to dataframe df
df = spark.read.format("csv").option("inferSchema", "true").option("header",
"true").option("delimiter",",").load(file_location)
 
#display the dataframe
display(df)

Create a table on top of the data in the data lake

In the previous section, we loaded the data from a CSV file to a DataFrame so that it can be accessed using python spark API.  Now, we will create a Hive table in spark with data in an external location (ADLS), so that the data can be access using SQL instead of python code.

In a new cell, copy the following command:

%sql
CREATE DATABASE covid_researc

Next, create the table pointing to the proper location in the data lake.

%sql
CREATE TABLE IF NOT EXISTS covid_research.covid_data
USING CSV
LOCATION 'abfss://[email protected]/raw/covid
19/johns-hopkins-covid-19-daily-dashboard-cases-by-states.csv'

 You should see the table appear in the data tab on the left-hand navigation pane.

Run a select statement against the table.

%sql
CREATE TABLE IF NOT EXISTS covid_research.covid_data
USING CSV
LOCATION 'abfss://[email protected]/raw/covid1
9/johns-hopkins-covid-19-daily-dashboard-cases-by-states.csv'
OPTIONS (header "true", inferSchema "true")

That concludes our step-by-step guide on accessing Azure Data Lake Storage Gen2 in Databricks, using the ADLS Gen2 storage account access key directly.

Hope you found this guide useful, stay tuned for more.

The scalability, flexibility, cost-efficiency, and improved performance that comes with moving to the cloud is becoming too attractive for companies to ignore them. Many organizations have started moving to the cloud as a cost-effective option to manage their IT portfolio and avoid expensive Capex for the purchase of new servers and remove the complexities in managing on-premises architecture.

Irrespective of a company’s size, migrating to the cloud is certainly quite an undertaking. Fortunately, Microsoft has created a unique platform with a range of tools to help make the migration fast and smooth, while minimizing the risk and impact to your business.

Microsoft Azure is one of the leading cloud computing service providers that allows businesses to use cloud resources on a pay per use model, therefore you can pay for only what you need and how long you need it. Azure provides multiple options right from Infrastructure as a Service (IaaS) to Platform as a Service (PaaS) to Software as a Service (SaaS), so you can choose from a simple lift and shift approach to a more complex application modernization approach.

The migration journey begins with an Assessment of your current setup. There are tools like Azure Migrate that Azure provides for this purpose and additionally you can leverage other assessment tools from Azure migration partner ecosystem. Azure Migrate provides a centralized hub to assess and migrate to Azure on-premise servers, infrastructures, applications, and data. With Azure, you can also assess how your workloads will perform, plan, and implement your migration strategy accordingly.

5 Azure Migration Strategies

Here are five strategies that are adopted widely for migrating an application to Azure cloud.

1. Rehosting

Commonly known as “lift and shift”, this is an approach to migrate applications from an on-premise environment to the cloud with no changes to the underlying applications. This is the most popular migration approach as it allows quick migration with little risk of disruption by employing real-time replication during the transition process.

2. Refactoring

This is also known as “repacking”. It involves making small changes to the code and configuration of the application to ensure they are more compatible with the cloud so you can connect them easily to Azure-native infrastructure. This can improve the scalability and maximize the operational cost-efficiency of the platform.

3. Rearchitecting

Also known as “redesigning”, this strategy involves modifying or extending the code base of an application to optimize it to run on Azure. Rearchitecting is a time-consuming migration approach, but still, it offers infinite scalability.

4. Rebuilding

This strategy involves discarding the old application and rebuilding an application or workload from the ground up using the Azure Platform as a Service (PaaS). In this migration strategy, you manage the applications and services you develop, while Azure manages the platform and infrastructure required to run it.

5. Replacing

Under this approach, all the underlying infrastructure, middleware, application software, and application data are in the cloud and managed by Azure in Microsoft datacenters. This is used for greater efficiency and scalability.

3-Step Migration Process

Once you decide on your migration approach, the actual migration to the cloud is a 3-step process (Assess – Migrate – Optimize). Now before you get started with the migration there are a few preliminary considerations to ensure your cloud environment is ready to receive your workloads. You need to ensure that your virtual data center in the cloud contains the elements that are comparable to your on-premises environment. Building the virtual data center in the cloud is a streamlined process and it includes the following

1. Identity

To ensure authenticated access to users between your on-premises environment and workloads that you have migrated to the cloud, you need to invest in a built-in identity management solution. For this purpose, you can use the Azure Active Directory (Azure AD) or other similar solutions.

2. Storage

Migrating to the cloud requires a storage platform that meets the performance needs of your migrated workloads. You can choose from different storage types and configure exact storage requirements based on workloads to ensure security and reliability. You just need to enter a few details to get the right storage for your migration project.

3. Networking

Networking is the backbone of the data center. When migrating to the cloud, you need to keep the applications in the same subnets and IP address ranges to ensure a seamless migration.  You can create a virtual network to maintain the same performance and stability you had in the on-premise data center.

4. Connectivity

During migration, you’ll transfer a large amount of data to the cloud. So it would be wise to opt for a dedicated connectivity option to help with smooth data transfer and have the best user experience. For this purpose, you can use Azure ExpressRoute as it helps in a faster, private connection to Azure and ensures performance and security.

Now it’s time to begin your migration journey to the cloud.

Migration Phase 1: Assessment

Now that you have a better understanding of Azure and how it fits into your migration strategy, it’s time to assess your existing infrastructure. Here are four steps to do that

1. Identification of application and server dependencies

Begin with inventory and assessment of on-premises IT resources to identify opportunities to optimize the IT environment and prioritize which applications and workloads are ideal for migration. Determining your priorities and objectives early can help you have a seamless migration process.

2. Assessment of on-premises applications and servers

Your organization may run hundreds or thousands of servers and virtual machines. You need consolidated planning and a perfect tool to shift them to the cloud. Microsoft offers Azure Migrate service to provide automation for the assessment of on-premises workloads. Ultimately, the goal of this assessment phase is to collect server and application information, including configuration and usage.

3. Configuration analysis

Configuration analysis will help you understand which of your workloads can be migrated with no modifications, which ones require a few modifications, and which workloads are incompatible with the current installation. Essentially this step helps you ensure the proper functioning of the workloads on the cloud.

4. Cost planning

The final step of the assessment phase is to collect resource usage such as CPU, memory, and storage to forecast costs and expenditures. This helps in ascertaining the actual usage of your workload and ensure that your choice meets both performance and economic targets.

Migration Phase 2: Execute Migration

After you’ve completed discovery and assessment, now it’s time to prepare for the next step – migration.  The lift-and-shift method most often employed for server or VM migration is real-time replication, due to its flexibility and capability in staged migration.

1. Real-time Replication

This involves creating a copy of the workload in the cloud and allowing asynchronous replication to keep the copy and the workload in sync. Replication also lets groups of virtual machines be connected to the cloud. Real-time replication also allows the old workload to remain online and accessible during the migration to ensure zero disruptions.

2. Testing

Once the replication is complete, start your application or workloads using an isolated environment that mimics the cloud production environment. It lets you test the application without impacting the on-premise as well as cloud production systems. When you’re fully satisfied, it’s time to perform the final migration.

Migration Phase 3: Optimize

Once the migration phase is complete, you need to ensure a seamless transition of operating workloads in the cloud. This is what the optimize phase is all about.

1. Secure cloud resources

Know the security controls and the capabilities of the new cloud-based application, to ensure that the security measures are working, and responding properly. You should become familiar with the capabilities of the Azure Security Center like centralized policy management, continuous security assessment, actionable recommendations, and more.

2. Protecting Data

Ensure that the workloads and data are having a proper backup, disaster recovery, encryption, and other measures to protect your business from risks. Azure offers multiple mechanisms like Azure disk encryption, Azure Backup, and Azure Site Recovery to protect your data.

3. Monitoring Cloud Health

Azure offers many monitoring services to ensure you have full visibility into your current system status and get unique insights into your applications and infrastructure. The basic monitoring services include Azure Monitor, Service Health, and Azure Advisor. A few of the premium monitoring services include Application Insights, Azure Log Analytics, and Network Watcher.

There are many options and reasons for migrating workloads to Azure.

With this straightforward guide, migration to Azure wouldn’t be a complex task anymore. By having a proper plan and mapping out the key objectives, you can ensure a successful Azure migration.

Want to know the key benefits of using Windows Server and SQL Server with Microsoft Azure? Here are four of them right from reducing costs and streamlining IT resources; modernizing by migrating to a flexible, open cloud; innovating new apps or managing existing server apps with unlimited flexibility; and ensuring data protection, security, and business continuity.

Would you like to modernize digital processes to improve profitability and ensure data security? At CloudIQ Technologies, we have knowledgeable and professional team ready to help you. Contact us today to learn more.

Azure Databricks provides comprehensive end-to-end diagnostic logs of activities performed by Azure Databricks users, allowing your enterprise to monitor detailed Azure Databricks usage patterns.

In this article, we’re going to look at sending the logs of Azure Databricks workspace to log analytics workspace using diagnostics settings present in the Databricks workspace.

Here are the pre-requisites and steps to enable diagnostics setting for Azure Databricks

Pre-requisites:

1. User with owner or contributor access where the Databricks workspace is deployed.
2. Databricks workspace. The diagnostics logging for Azure Databricks service is available only for the Premium plan.

Steps:

1. Login to Azure portal.
2. Select the Databricks workspace.

3. Select the diagnostics settings.

4. Now click “+ Add Diagnostics Settings”.

5. Azure Databricks provides diagnostic logs for the following services:

• DBFS
• Clusters
• Pools
• Accounts
• Jobs
• Notebook
• SSH
• Workspace
• Secrets
• SQL Permissions

6. Here we are going to send the logs to the log analytics workspace.

7. Select all the logs you want and send them to log analytics. Here we’re sending cluster logs.

8. Click Save.

9. Allow some time to ingest the logs to log analytics workspace.

10. Now go to the log analytics workspace where the diagnostics are configured.

11. Select logs. Now using KQL we can query our data sent from the Databricks workspace.

12. The Databricks log tables are found under the LogManagement category.

Databricks Monitoring Dashboard

Here is the simple Databricks Monitoring dashboard we created for

• Cluster availability
• Failed job trend
• Success vs failed job trend

We hope this article helps you set up the right configurations to sending the logs of Azure Databricks workspace to log analytics workspace and build the Databricks monitoring dashboard.

Did you know migrating to Microsoft Azure can reduce your data center footprint 73%? 

Take advantage of your current investments and IT skills in Microsoft technologies. Microsoft applications and Azure have been built to work better together with flexibility, high compatibility and hybrid capabilities. 

Check out this infographic to learn, how you can get unparalleled cost savings, easily plan migrations, avoid complexity of multi-vendor support, and modernize your applications in the cloud from the leader you already trust.

Migrate to Azure at your own pace with confidence and support from CloudIQ. At CloudIQ Technologies, we have knowledgeable and professional team ready to help you modernize workloads with Azure. Contact us today to learn more.

test beta

Break down the cloud journey with four stages of the process—starting with a pre-migration assessment and then looking at migration, post-migration, and optimization. Microsoft Azure has you covered with tools created specifically for you.

Would you like to modernize your apps and data on Azure? At CloudIQ Technologies, we have knowledgeable and professional team ready to help you. Contact us today to learn more.

It’s time to create and implement business and technology strategies powered by the cloud. Here is your complete game plan. Check the “Cloud Adoption Framework” infographic to plan your strategy to modernize and innovate. 

Would you like to migrate to the cloud? At CloudIQ Technologies, we have knowledgeable and professional team ready to help you successfully adopt Cloud. Contact us today to learn more.

In our previous blog on getting started with Azure Databricks, we looked at Databricks tables.  In this blog, we will look at a type of Databricks table called Delta table and best practices around storing data in Delta tables.

1. Delta Lake

Delta Lake is an open-source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs. Databricks Delta table is a table that has a Delta Lake as the data source similar to how we had a CSV file as a data source for the table in the previous blog.

2. Table which is not partitioned

When we create a delta table and insert records into it, Databricks loads the data into multiple small files.  You can see the multiple files created for the table “business.inventory” below

3. Partitioned table

Partitioning involves putting different rows into different tables.  E.g., if we have an address table with addresses in the US, the addresses might be stored in 50 different tables corresponding to the 50 states in the US.  A view with a union might be created over all of them to provide a complete view of all addresses.

Sample code to create a table partitioned by date column is given below:

CREATE TABLE events (
  date DATE,
  eventId STRING,
  eventType STRING,
  data STRING)
USING delta
PARTITIONED BY (date) 

The table “business.sales” given below is partitioned by InvoiceDate.  You can see that there is a folder created for each InvoiceDate and within the folders, there are multiple files that store the data for this table.

This partitioning will be useful when we have queries selecting records from this table with InvoiveDate in WHERE clause. 

E.g.:
SELECT SLSDA_ID, RcdType, DistId
FROM business.sales
WHERE InvoiceDate = ‘2013-01-01’

In total there are 40,545 files for this table which you can see from below screenshot

4. OPTIMIZE

SMALL FILE PROBLEM

Historical and new data is often written in very small files and directories.  This data may be spread across a data center or even across the world (that is, not co-located).  The result is that a query on this data may be very slow due to

• network latency
• volume of file metadata

The solution is to compact many small files into one larger file.

OPTIMIZE command invokes the bin-packing (Compaction) algorithm to coalesce small files into larger ones.  Small files are compacted together into new larger files up to 1GB.

You can see below that the OPTIMIZE command has removed the 40,545 files and instead of them added 2378 files.  Also, observe that after Optimization the size of the table has decreased from 1.49 GB to 1.08 GB

5. Optimize table which is not partitioned

Optimize will compact the small files for tables that are not partitioned too.

business.finance_transactions_silver table is not partitioned and is currently having 64 files with total size 858 MB

Running the Optimize command coalesces the 64 files to 1 file

Note that “partitionsOptimized” is 1 in this case.  Previously for the partitioned table “partitionsOptimized was 2509.  OPTIMIZE command coalesces the small files within a partition only.  If the table is not partitioned, the whole table is considered as a single xpartition.

6. ZORDER

• Data Skipping is a performance optimization that aims at speeding up queries that contain filters (WHERE clauses).
  As new data is inserted into a Databricks Delta table, file-level min/max statistics are collected for all columns (including nested ones) of supported types. Then, when there’s a lookup query against the table, Databricks Delta first consults these statistics to determine which files can safely be skipped.  This is done automatically and no specific commands are required to be run for this.
• Z-Ordering is a technique to co-locate related information in the same set of files.
  Z-Ordering maps multidimensional data to one dimension while preserving the locality of the data points.

Given a column that you want to perform ZORDER on, say OrderColumn, Delta

• Takes existing parquet files within a partition.
• Maps the rows within the parquet files according to OrderColumn using the Z-order curve algorithm.
• In the case of only one column, the mapping above becomes a linear sort
• Rewrites the sorted data into new parquet files.

Note: We cannot use the table partition column also as a ZORDER column.

Syntax for ZORDER is

OPTIMIZE tablename
ZORDER BY (OrderColumn) 

7. Best practices

a. PARTITION BY

• Partition the table by a column which is used in the WHERE clause or ON clause (join).  The most commonly used partition column is the date.
• Use columns with low cardinality.  If the cardinality of a column will be very high, do not use that column for partitioning. For example, if you partition by a column userId and if there can be 1M distinct user IDs, then that is a bad partitioning strategy.
• Amount of data in each partition: You can partition by a column if you expect data in that partition to be at least 1 GB.  Partitioning is not required for smaller tables.
• PARTITION BY is done on a single column only

b. OPTIMIZE

• OPTIMIZE is required for all tables to which we write data continuously on a daily basis.
• OPTIMIZE is not required for tables that have static data/reference data which are rarely updated.
• There is a cost associated with OPTIMIZE (Running Optimize command for sales took 6.64 minutes).  We should run it more often (daily) if we want better end-user query performance.  We should run it less often if we want to optimize costs.

c. ZORDER BY

• If we expect a column to be commonly used in query predicates and if that column has high cardinality (that is, a large number of distinct values), then use ZORDER BY.
• We can specify multiple columns for ZORDER BY as a comma-separated list. However, the effectiveness of the locality drops with each additional column.

8. References:

• Partition – https://docs.databricks.com/delta/quick-start.html#partition-data&language-sql
• Optimize, Data skipping and Z-Ordering – https://docs.databricks.com/delta/optimizations/file-mgmt.html
  This page gives guidelines on which column can be used for ZORDER BY.  FAQ at bottom of this website tells how often we should run OPTIMIZE
• https://docs.databricks.com/delta/best-practices.html – gives guidelines on how to choose the right partition column.

Migrating your IT infrastructure to the cloud has a ton of benefits. Whether you’re looking to improve security and become GDPR compliant, cut your total cost of ownership, or promote teamwork and innovation by integrating AI capabilities, the cloud provides a solution to your IT problems.

Would you like to upgrade your IT infrastructure to the cloud? At CloudIQ Technologies, we have knowledgeable and professional team ready to help address any of your IT infrastructure upgrade needs. Contact us today to learn more.

Backing up on-premises resources to the cloud leverages the power and scale of the cloud to deliver high-availability with no maintenance or monitoring overhead. With Azure Backup service the benefits keep adding up, right from data security to centralized monitoring and management.

Azure Backup service uses the MARS agent to back up files, folders, and system state from on-premises machines and Azure VMs. The backups are stored in a Recovery Services vault.

In this article, we will look at how to back up on-premise files and folders using Microsoft Azure Recovery Services (MARS) agent.

There are two ways to run the MARS agent:

• Directly on on-premises Windows machines.
• On Azure VMs that run Windows side by side with the Azure VM backup extension.

Here is the step by step process.

Create a Recovery Services vault

1. Sign in to the Azure portal.
2. On the Recovery Services vaults dashboard, select “Add”.

3. The Recovery Services vault dialog box opens. Provide values of the Name, Subscription, Resource group, and Location.
4. Select “Create”.

Download the MARS agent

1. Download the MARS agent so that you can install it on the machines that you want to back up.
2. In the vault, select “Backup”.
3. Select On-premises for “Where is your workload running?”
4. Select Files and folders for “What do you want to back up?”
5. Select “Prepare Infrastructure”.

5. For “Prepare infrastructure”, under Install Recovery Services agent, download the MARS agent.
6. Select “Already downloaded or using the latest Recovery Services Agent”, and then download the vault credentials.
7. Select “Save”.

Install and register the agent

1. Run the MARSagentinstaller.exe file on the VM.
2. In the “MARS Agent Setup Wizard”, select “Installation Settings”.
3. Choose where to install the agent and choose a location for the cache. Select “Next”.

• The cache is for storing data snapshots before sending them to recovery services vault.
• The cache location should have free space equal to at least 5 percent of the size of the data you’ll back up.

4. For Proxy Configuration, specify how the agent that runs on the Windows machine will connect to the internet. Then select “Next”.

5. For Installation, review, and select “Install”.
6. After the agent is installed, select “Proceed to Registration”.

7. In Register Server Wizard > Vault Identification, browse to and select the credentials file. Then select “Next”.

8. On the “Encryption Setting” page, specify a passphrase(user-defined) which is used to encrypt and decrypt backups for the machine.

9. Save the passphrase in a secure location. It is needed while restoring a backup.
10. Select “Finish”.

Create a backup policy

Steps to create a backup policy:

1. Open the MARS agent console.
2. Under “Actions”, select “Schedule Backup”.

3. In the “Schedule Backup Wizard”, select “Getting started” and click “Next”.
4. Under “Select Items to Back up”, select “Add Items”.

5. Select items to back up, and select OK.

6. On the “Select Items to Back Up” page, select “Next”.
7. Specify when to take daily or weekly backups in the “Specify Backup Schedule” page and select “Next”.
8. It is possible to schedule up to three daily backups per day and can run weekly backups too.

9. On the “Select Retention Policy” page, specify how to store copies of your data. And select “Next”.
10. On the Confirmation page, review the information, and then select “Finish”.

11. After the wizard finishes creating the backup schedule, select “Close”.

We hope this step by step guide helps you back up on-premise files and folders using Microsoft Azure Recovery Services (MARS) agent.

Many businesses are struggling to find the talent and capacity to create and manage their machine learning models to actually unlock the insights within their data. Here is an infographic that shows how Microsoft Azure Machine Learning streamlines this process to make modeling accessible to all businesses. 

What’s holding your business back from using AI to turn your data into actionable insights? Azure Machine Learning makes AI more accessible to businesses of all sizes and experience levels by reducing cost and helping you create and manage your models. But you don’t have to go it alone. We can help you assess your business needs and adopt the right AI solution. Contact us today to learn how we can help transform your business with AI.

The distributed nature of cloud applications requires a messaging infrastructure that connects the components and services, ideally in a loosely coupled manner in order to maximize scalability. In this article let’s explore the asynchronous messaging options in Azure.

At an architectural level, a message is a datagram created by an entity (producer), to distribute information so that other entities (consumers) can be aware and act accordingly. The producer and the consumer can communicate directly or optionally through an intermediary entity (message broker).  

Messages can be classified into two main categories. If the producer expects an action from the consumer, that message is a command. If the message informs the consumer that an action has taken place, then the message is an event.

Commands

The producer sends a command with the intent that the consumer(s) will perform an operation within the scope of a business transaction.

A command is a high-value message and must be delivered at least once. If a command is lost, the entire business transaction might fail. Also, a command shouldn’t be processed more than once. Doing so might cause an erroneous transaction. A customer might get duplicate orders or billed twice.

Commands are often used to manage the workflow of a multistep business transaction. Depending on the business logic, the producer may expect the consumer to acknowledge the message and report the results of the operation. Based on that result, the producer may choose an appropriate course of action.

Events

An event is a type of message that a producer raises to announce facts.

The producer (known as the publisher in this context) has no expectations that the events will result in any action.

Interested consumer(s), can subscribe, listen for events, and take actions depending on their consumption scenario. Events can have multiple subscribers or no subscribers at all. Two different subscribers can react to an event with different actions and not be aware of one another.

The producer and consumer are loosely coupled and managed independently. The consumer isn’t expected to acknowledge the event back to the producer. A consumer that is no longer interested in the events, can unsubscribe. The consumer is removed from the pipeline without affecting the producer or the overall functionality of the system.

There are two categories of events:

• The producer raises events to announce discrete facts. A common use case is event notification. For example, Azure Resource Manager raises events when it creates, modifies, or deletes resources. A subscriber of those events could be a Logic App that sends alert emails.
  
• The producer raises related events in a sequence, or a stream of events, over a period of time. Typically, a stream is consumed for statistical evaluation. The evaluation can be done within a temporal window or as events arrive. Telemetry is a common use case, for example, health and load monitoring of a system. Another case is event streaming from IoT devices.

A common pattern for implementing event messaging is the Publisher-Subscriber pattern.

Role and benefits of a message broker

An intermediate message broker provides the functionality of moving messages from producer to consumer and can offer additional benefits.

Decoupling

A message broker decouples the producer from the consumer in the logic that generates and uses the messages, respectively. In a complex workflow, the broker can encourage business operations to be decoupled and help coordinate the workflow.

Load balancing

Producers may post a large number of messages that are serviced by many consumers. Use a message broker to distribute processing across servers and improve throughput. Consumers can run on different servers to spread the load. Consumers can be added dynamically to scale out the system when needed or removed otherwise.

Load leveling

The volume of messages generated by the producer or a group of producers can be variable. At times there might be a large volume causing spikes in messages. Instead of adding consumers to handle this work, a message broker can act as a buffer, and consumers gradually drain messages at their own pace without stressing the system.

Reliable messaging

A message broker helps ensure that messages aren’t lost even if communication fails between the producer and consumer. The producer can post messages to the message broker and the consumer can retrieve them when communication is re-established. The producer isn’t blocked unless it loses connectivity with the message broker.

Resilient messaging

A message broker can add resiliency to the consumers in your system. If a consumer fails while processing a message, another instance of the consumer can process that message. The reprocessing is possible because the message persists in the broker.

Technology choices for a message broker

Azure provides several message broker services, each with a range of features.

Azure Service Bus

Azure Service Bus queues are well suited for transferring commands from producers to consumers. Here are some considerations.

Pull model

A consumer of a Service Bus queue constantly polls Service Bus to check if new messages are available. The client SDKs and Azure Functions trigger for Service Bus abstract that model. When a new message is available, the consumer’s callback is invoked, and the message is sent to the consumer.

Guaranteed delivery

Service Bus allows a consumer to peek the queue and lock a message from other consumers.

It’s the responsibility of the consumer to report the processing status of the message. Only when the consumer marks the message as consumed, Service Bus removes the message from the queue. If a failure, timeout, or crash occurs, Service Bus unlocks the message so that other consumers can retrieve it. This way messages aren’t lost in the transfer.

Message ordering

If you want consumers to get the messages in the order they are sent, Service Bus queues guarantee first-in-first-out (FIFO) ordered delivery by using sessions. A session can have one or more messages.

Message persistence

Service bus queues support temporal decoupling. Even when a consumer isn’t available or unable to process the message, it remains in the queue.

Checkpoint long-running transactions

Business transactions can run for a long time. Each operation in the transaction can have multiple messages. Use checkpointing to coordinate the workflow and provide resiliency in case a transaction fails.

Hybrid solution

Service Bus bridges on-premises systems and cloud solutions. On-premises systems are often difficult to reach because of firewall restrictions. Both the producer and consumer (either can be on-premises or the cloud) can use the Service Bus queue endpoint as the pickup and drop off location for messages.

Topics and subscriptions

Service Bus supports the Publisher-Subscriber pattern through Service Bus topics and subscriptions.

Azure Event Grid

Azure Event Grid is recommended for discrete events. Event Grid follows the Publisher-Subscriber pattern. When event sources trigger events, they are published to Event grid topics. Consumers of those events create Event Grid subscriptions by specifying event types and an event handler that will process the events. If there are no subscribers, the events are discarded. Each event can have multiple subscriptions.

Push Model

Event Grid propagates messages to the subscribers in a push model. Suppose you have an event grid subscription with a webhook. When a new event arrives, Event Grid posts the event to the webhook endpoint.

Custom topics

Create custom Event Grid topics, if you want to send events from your application or an Azure service that isn’t integrated with Event Grid.

High throughput

Event Grid can route 10,000,000 events per second per region. The first 100,000 operations per month are free.

Resilient delivery

Even though successful delivery for events isn’t as crucial as commands, you might still want some guarantee depending on the type of event. Event Grid offers features that you can enable and customize, such as retry policies, expiration time, and dead lettering.

Azure Event Hubs

When working with an event stream, Azure Event Hubs is the recommended message broker. Essentially, it’s a large buffer that’s capable of receiving large volumes of data with low latency. The data received can be read quickly through concurrent operations. You can transform the data received by using any real-time analytics provider. Event Hubs also provides the capability to store events in a storage account.

Fast ingestion

Event Hubs are capable of ingesting millions of events per second. The events are only appended to the stream and are ordered by time.

Pull model

Like Event Grid, Event Hubs also offers Publisher-Subscriber capabilities. A key difference between Event Grid and Event Hubs is in the way event data is made available to the subscribers. Event Grid pushes the ingested data to the subscribers whereas Event Hub makes the data available in a pull model. As events are received, Event Hubs appends them to the stream. A subscriber manages its cursor and can move forward and back in the stream, select a time offset, and replay a sequence at its pace.

Partitioning

A partition is a portion of the event stream. The events are divided by using a partition key. For example, several IoT devices send device data to an event hub. The partition key is the device identifier. As events are ingested, Event Hubs move them to separate partitions. Within each partition, all events are ordered by time.

Event Hubs Capture

The Capture feature allows you to store the event stream to Azure Blob storage or Data Lake Storage. This way of storing events is reliable because even if the storage account isn’t available, Capture keeps your data for a period, and then writes to the storage after it’s available.

We hope this quick start guide helps you get stated on azure messaging and event driven architecture.

Azure Databricks lets you spin up clusters and build quickly in a fully managed Apache Spark environment with the global scale and availability of Azure. And of course, for any production-level solution, monitoring is a critical aspect.

Azure Databricks comes with robust monitoring capabilities for custom application metrics, streaming query events, and application log messages. It allows you to push this monitoring data to different logging services.

In this article, we will look at the setup required to send application logs and metrics from Microsoft Azure Databricks to a Log Analytics workspace.

Prerequisites

1. Clone the repository mentioned below
   https://github.com/mspnp/spark-monitoring.git
   
2. Azure Databricks workspace
3. Azure Databricks CLI
   Databricks workspace personal access token is required to use the CLI
   You can also use the Databricks CLI from Azure Cloud Shell.
   
4. Java IDEs with the following resources
   Java Development Kit (JDK) version 1.8
   Scala language SDK 2.11
   Apache Maven 3.5.4

Building the Azure Databricks monitoring library with Docker

After cloning repository please open the terminal in the respective path

Please run the command as follows
Windows :

docker run -it --rm -v %cd%/spark-monitoring:/spark-monitoring -v "%USERPROFILE%/.m2":/root/.m2 maven:3.6.1-jdk-8 /spark-monitoring/build.sh 

Linux:

chmod +x spark-monitoring/build.sh
docker run -it --rm -v `pwd`/spark-monitoring:/spark-monitoring -v "$HOME/.m2":/root/.m2 maven:3.6.1-jdk-8 /spark-monitoring/build.sh 

Configuring Databricks workspace

dbfs configure –token
It will ask for Databricks workspace URL and Token
Use the personal access token that was generated when setting up the prerequisites
You can get the URL from
Azure portal > Databricks service > Overview

 dbfs mkdirs dbfs:/databricks/spark-monitoring 

Open the file /src/spark-listeners/scripts/spark-monitoring.sh
Now add the Log Analytics  Workspace ID and Key

Use Databricks CLI to copy the modified script

dbfs cp <local path to spark-monitoring.sh> dbfs:/databricks/spark-monitoring/spark-monitoring.sh 

Use Databricks CLI to copy all JAR files generated

dbfs cp --overwrite --recursive <local path to target folder> dbfs:/databricks/spark-monitoring/ 

Create and configure the Azure Databricks cluster

1. Navigate to your Azure Databricks workspace in the Azure Portal.
2. On the home page, click on “new cluster”.
3. Choose a name for your cluster and enter it in the text box titled “cluster name”.
4. In the “Databricks Runtime Version” dropdown, select 5.0 or later (includes Apache Spark 2.4.0, Scala 2.11).

6 Click the “create cluster” button to create the cluster. Next, click on the “start” button to start the cluster.

Now you can run the jobs in the cluster and can get the logs in the Log Analytics workspace

We hope this article helps you set up the right configurations to send application logs and metrics from Azure Databricks to your Log Analytics workspace.

This infographic outlines a day in the life of a remote worker using Microsoft Teams to collaborate, create, and be more productive while working at home. See how this individual uses multiple features to stay connected with the team and work efficiently

The daily life of most workers has changed drastically as COVID19 has made employees move to home offices. But your team can still get work done. Microsoft Teams can make it possible. Contact us to enable MS teams for your organization.

Databricks is a web-based platform for working with Apache Spark, that provides automated cluster management and IPython-style notebooks.  To understand the basics of Apache Spark, refer to our earlier blog on how Apache Spark works

Databricks is currently available on Microsoft Azure and Amazon AWS.  In this blog, we will look at some of the components in Azure Databricks.

1.   Workspace

A Databricks Workspace is an environment for accessing all Databricks assets. The Workspace organizes objects (notebooks, libraries, and experiments) into folders, and provides access to data and computational resources such as clusters and jobs.

Create a Databricks workspace

The first step to using Azure Databricks is to create and deploy a Databricks workspace. You can do this in the Azure portal.

1. In the Azure portal, select Create a resource > Analytics > Azure Databricks.
2. Under Azure Databricks Service, provide the values to create a Databricks workspace.
   
   a. Workspace Name: Provide a name for your workspace.
   b. Subscription: Choose the Azure subscription in which to deploy the workspace.
   c. Resource Group: Choose the Azure resource group to be used.
   d. Location: Select the Azure location near you for deployment.
   e. Pricing Tier: Standard or Premium

Once the Azure Databricks service is created, you will get the screen given below.  Clicking on the Launch Workspace button will open the workspace in a new tab of the browser.

2.   Cluster

A Databricks cluster is a set of computation resources and configurations on which we can run data engineering, data science, and data analytics workloads, such as production ETL pipelines, streaming analytics, ad-hoc analytics, and machine learning.

To create a new cluster:

1. Select Clusters from the left-hand menu of Databricks’ workspace.
2. Select Create Cluster to add a new cluster.

We can select the Scala and Spark versions by selecting the appropriate Databricks Runtime Version while creating the cluster.

3.   Notebooks

A notebook is a web-based interface to a document that contains runnable code, visualizations, and narrative text.  We can create a new notebook using either the “Create a Blank Notebook” link in the Workspace (or) by selecting a folder in the workspace and then using the Create >> Notebook menu option.

While creating the notebook, we must select a cluster to which the notebook is to be attached and also select a programming language for the notebook – Python, Scala, SQL, and R are the languages supported in Databricks notebooks.

The workspace menu also provides us the option to import a notebook, by uploading a file (or) specifying a file.  This is helpful if we want to import (Python / Scala) code developed in another IDE (or) if we must import code from an online source control system like git.

In the below notebook we have python code executed in cells Cmd 2 and Cmd 3; a python spark code executed in Cmd 4.  The first cell (Cmd 1) is a Markdown cell.  It displays text which has been formatted using markdown language.

Magic commands

Even though the above notebook was created with Language as python, each cell can have code in a different language using a magic command at the beginning of the cell.  The markdown cell above has the code below where %md is the magic command:

%md Sample Databricks Notebook 

The following provides the list of supported magic commands:

• %python – Allows us to execute Python code in the cell.
• %r – Allows us to execute R code in the cell.
• %scala – Allows us to execute Scala code in the cell.
• %sql – Allows us to execute SQL statements in the cell.
• %sh – Allows us to execute Bash Shell commands and code in the cell.
• %fs – Allows us to execute Databricks Filesystem commands in the cell.
• %md – Allows us to render Markdown syntax as formatted content in the cell.
• %run – Allows us to run another notebook from a cell in the current notebook.

4.   Libraries

To make third-party or locally built code available (like .jar files) to notebooks and jobs running on our clusters, we can install a library. Libraries can be written in Python, Java, Scala, and R. We can upload Java, Scala, and Python libraries and point to external packages in PyPI, or Maven.

To install a library on a cluster, select the cluster going through the Clusters option in the left-side menu and then go to the Libraries tab.

Clicking on the “Install New” option provides us with all the options available for installing a library.  We can install the library either uploading it as a Jar file or getting it from a file in DBFS (Data Bricks File System).  We can also instruct Databricks to pull the library from Maven or PyPI repository by providing the coordinates.

5. Jobs

During code development, notebooks are run interactively in the notebook UI.  A job is another way of running a notebook or JAR either immediately or on a scheduled basis.

We can create a job by selecting Jobs from the left-side menu and then provide the name of job, notebook to be run, schedule of the job (daily, hourly, etc.)

Once the jobs are scheduled, the jobs can be monitored using the same Jobs menu.

6.   Databases and tables

A Databricks database is a collection of tables. A Databricks table is a collection of structured data. Tables are equivalent to Apache Spark DataFrames. We can cache, filter, and perform any operations supported by DataFrames on tables. You can query tables with Spark APIs and Spark SQL.

Databricks provides us the option to create new Tables by uploading CSV files; Databricks can even infer the data type of the columns in the CSV file.

All the databases and tables created either by uploading files (or) through Spark programs can be viewed using the Data menu option in Databricks workspace and these tables can be queried using SQL notebooks.

We hope this article helps you getting started with Azure Databricks. You can now spin up clusters and build quickly in a fully managed Apache Spark environment with the global scale and availability of Azure.

To keep business critical applications running 24/7/365 it is important for organizations to have a sound business continuity and disaster recovery strategy. In this article we will discuss how to set up disaster recovery for Azure VM in secondary region.

We will use Azure Site Recovery that helps manage and orchestrate disaster recovery of on-premises machines and Azure virtual machines (VM), including replication, failover, and recovery.

Pre-requisites,

1. Recovery services vault
2. A Virtual machine

ENABLING REPLICATION

To replicate a VM to secondary region, prepare site recovery infrastructure. In this case we are replicating from one azure region to another.

For replication from Azure to Azure, one can directly go to recovery service vault and replicate the VM.

1. Go to recovery services vault >> Replicated Items
2. Click on the Replicate icon and follow.

3. Select the following

• Source: Azure
• Source location: Region where the VM is deployed
• Azure virtual machine deployment model: Resource manager
• Source subscription: Subscription where the VM is deployed
• Source Resource group: Resource group where the VM is deployed

4. Select OK to proceed to next step.

5.Select the VM to replicate

6. Target configurations

• Target location: Secondary region where the VM is to be replicated
• Target subscription: Subscription where the VM to be replicated

By default, the following resources are created in target region and can be customized per your need

• Resource group
• Virtual network
• Cache storage account
• Replica managed disks
• Target availability sets (if applicable)

You can set replication policies and view extension details here

7.Click on create target resources

8.Then select Enable replication

Go to recovery services vault >> Monitoring >> Site recovery Jobs to view the jobs that are running during the replication.

Look for the following jobs

1. Prerequisites check for enabling protection
2. Installing Mobility Service and preparing target
3. Enable replication
4. Starting initial replication
5. Updating the provider states

Once the above-mentioned jobs are over, here is what happens

• Synchronization process begins
• Waiting for first recovery point
• The VM is protected

Once all processes are completed, you can view the VM by going to recovery services vault >> Replicated Items

Introduction to Terraform

Terraform is an open-source tool for managing cloud infrastructure. Terraform uses Infrastructure as Code (IaC) for building, changing and versioning infrastructure safely. Terraform is used to create, manage, and update infrastructure resources such as virtual machines, virtual networks, and clusters.

The Terraform CLI provides a simple mechanism to deploy and version the configuration files to Azure. And with AzureRM you can create, modify and delete azure resources in Terraform configuration.

The infrastructure that Terraform can manage, includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc.

Providers in Terraform

A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS

• Azure
• Aws
• Google Cloud
• OpenStack
• Docker
• Alibaba Cloud
• VMware   

For each provider, there are many kinds of resourcesyou can create. Here is the general Syntax for terraform resources.

resource  “<provider>_<type>”   “<name>” 	{
[config]
}

Where PROVIDER is the name of a provider (e.g., Azure), TYPE is the type of resources to create in that provider (e.g., Instance), NAME is an identifier you can use throughout the Terraform code to refer to this resource and CONFIG consists of one or more argumentsthat are specific to that resource.

Terraform Features:

Infrastructure as Code

Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used.

Execution Plans

Terraform has a “planning” step where it generates an execution plan. The execution plan shows what Terraform will do when you call apply. This lets you avoid any surprises when Terraform manipulates infrastructure.

Resource Graph

Terraform builds a graph of all your resources and parallelizes the creation and modification of any non-dependent resources. Because of this, Terraform builds infrastructure as efficiently as possible, and operators get insight into dependencies in their infrastructure.

Change Automation

Complex changesets can be applied to your infrastructure with minimal human interaction. With the previously mentioned execution plan and resource graph, you know exactly what Terraform will change and in what order, avoiding many possible human errors.

TERRAFORM STRUCTURE

The primary module structure requirement is that a “root module” must exist. The root module is the directory that holds the Terraform configuration files that are applied to build your desired infrastructure. Any module should include, at a minimum, a “main.tf”, a “variables.tf” and “outputs.tf” file.

main.tf calls modules, locals, and data-sources to create all resources. If using nested modules to split up your infrastructure’s required resources, the “main.tf” file holds all your module blocks and any needed resources not contained within your nested modules.

variables.tf contains the input variable and output variable declarations.

outputs.tf tells Terraform what data is important. This data is outputted when “apply” is called and can be queried using the Terraform “output” command. It contains outputs from the resources created in main.tf.

TFVARS File – To persist variable values, create a file and assign variables within this file. Within the current directory, for all files that match terraform.tfvars or *.auto.tfvars, terraform automatically loads them to populate variables.

MODULES

Modules are subdirectories with self-contained Terraform code. A module is a container for multiple resources that are used together. The root module is the directory that holds the Terraform configuration files that are applied to build your desired infrastructure. The root module may call other modules and connect them by passing output values from one as input values of another.

In production, we may need to manage multiple environments, and different products with similar infrastructure. Writing code to manage each of these similar configurations increases redundancy in the code.  And finally, we need the capability to test different versions while keeping the production infrastructure stable.

Terraform provides modules that allow us to abstract away re-usable parts, which can be configured once, and used everywhere. Modules allow us to group resources, define input variables which are used to change resource configuration parameters and define output variables that other resources or modules can use.

Modules can also call other modules using a “module” block, but we recommend keeping the module tree relatively flat and using module composition as an alternative to a deeply nested tree of modules, because this makes the individual modules easier to re-use in different combinations.

Terraform Workflow :

There are steps to build infrastructure with terraform

• INIT
• Plan
• Apply
• Destroy

INIT

Initialize the Terraform configuration directory using Terraform “init”.

Init will create a hidden directory “.terraform” and download plugins as needed by the configuration. Init also configures the “-backend-config” option and can be used for partial backend configuration.

Command

terraform init -backend-config=”backend-dev.config”

backend-dev.config – This file contains the details shown in the screenshot below.

PLAN

The terraform plan command is used to create an execution plan. The plan will be used to see all the resources that are getting created/updated/deleted, before getting applied. Actual creation will happen in the “apply” command.

The var file given will define resources that are unique for each team.

Command:

terraform plan -var-file="parentvarvalues.tfvars"

This file includes all global variables and Azure subscription details.

APPLY

The Terraform “apply” command is used to apply changes in the configuration. You’ll notice that the “apply” command shows you the same “plan” output and asks you to confirm if you want to proceed with this plan.

The “-auto-approve” parameter will skip the confirmation for creating resources. It’s better not to have it when you want to apply directly, without “plan”.

Command:

terraform apply -var-file="parentvarvalues-team1.tfvars" -auto-approve

Terraform State Management:

Terraform stores the resources it manages into a state file. There are two types of state files: “remote” and “local”. While the “local” state is great for an isolated developer, the “remote” state is quite indispensable for a team, as each member will need to share the infrastructure state whenever there is a change.

Terraform compares those changes with the state file to determine what changes result in a new resource or resource modifications. Terraform stores the state about our managed infrastructure and configuration. This state is used by Terraform to map real-world resources to our configuration, keep track of metadata, and to improve performance for large infrastructures.

TERRAFORM IMPORT

The terraform import command is used to import existing infrastructure. This allows you to take resources you’ve created by some other means and bring it under Terraform management. This is a great way to slowly transition infrastructure to Terraform.

resource “azurerm_resourcegroup .name <subscription_id>{
#instance configuration
}

You want to import the state that already exists, so that the next time you the “apply” command, terraform already knows that the resource exists, and any changes made going forward will be picked up as modifications.

79% of analytics users encounter data questions they can’t solve each month. How are you helping your team to uncover insights from data?

Moving data isn’t always easy. There are more than 340 types of databases in use today and moving data across them presents challenges for any IT team. At CloudIQ Technologies, we have years of experience helping businesses find the IT solutions that can keep up with their constantly evolving business practice. Whether you need a solution for storage, data transfer, or just need to gain better insights from your data, we can help.

KEXP is known internationally for their music and authenticity. To help bring their global audience the music they want, KEXP needed to find a solution that could bring all of their services online. While they eventually accomplished their goal, they did run into roadblocks along the way.

Your partners at CloudIQ Technologies and Microsoft can help you overcome any obstacle. With years of industry experience, you can be rest assured knowing your custom IT solution will be up and running in no time. Contact us today to find out more on how we can help.

Cloud security is a major challenge for organizations running mission critical applications on cloud. One of the biggest risks from hackers come via open ports, and Microsoft Azure Security Center provides a great option to manage this threat – Just-in-Time VM access!

What is Just-in-Time VM access?

With Just-in-Time VM access, you can define what VM and what ports can be opened and controlled and for how long. The Just-in-Time access locks down and limits the ports of Azure virtual machines in order to overcome malicious attacks on the virtual machine, therefore only providing access to a port for a limited amount of time. Basically, you block all inbound traffic at the network level.

When Just-In-Time access is enabled, every user’s request for access will be routed through Azure RBAC, and access will be granted only to users with the right credentials. Once a request is approved, the Security Center automatically configures the NSGs to allow inbound traffic to these ports – only for the requested amount of time, after which it restores the NSGs to their previous states.

The just-in-time option is available only for the standard security center tier and is only applicable for VMs deployed via Azure resource manager.

What are the permissions needed to configure and use JIT?

Why Just-in-Time access?

Consider the scenario where a virtual machine is deployed in Azure, and the management port is opened for all IP addresses all the time. This leaves the VM open for brute force attack.

The brute force attack is usually targeted to Management ports like SSH (22) and RDP (3389). If the attacker compromises the security, the whole VM will be open to them. Even though we might have NSG firewalls enabled in our Azure infrastructure, it’s best to limit the exposure of management port within the team for a limited amount of time.

How to enable JIT?

Just-In-Time access can be implemented in two ways,

1. Go to Azure security center and click on Just-in-Time VM

2. Go to the VM, then click on configuration and “Enable JIT”

How to set up port restrictions?

Go to Azure security center and click on recommendations for Compute &Apps.

Select the VM and click Enable JIT on VMs.

It will then show a list of recommended ports. It is possible to add additional ports as per requirement. The default port list is show below.

Now click on the port that you wish to restrict. A new tab will appear with information on the protocol to be allowed, allowed source IP (per IP address, or a CIDR range).

The main thing to note is the request time. The default time is 3 hours; it can be increased or decreased as per the requirement. Then click, OK.

Click OK and the VM will appear in the Just-in-Time access window in the security center.

What changes will happen in the infrastructure when JIT is enabled?

The Azure security center will create a new Deny rule with a priority less than the original Management port’s Allow rule in the Network security group’s Inbound security rule.

If the VM is behind an Azure firewall, the same rule overwrite occurs in the Azure firewall as well.

How to connect to the JIT enabled VM?

Go to Azure security center and navigate to Just-in-Time access.

Select the VM that you need to access and click on “Request Access”

This will take you to the next page where extra details need to be provided for connectivity such as,

1. Click ON Toggle
2. Provide Allowed IP ranges
3. Select time range
4. Provide a justification for VM Access
5. Click on Open Ports

This process will overwrite the NSG Deny rule and create a new Allow rule with less priority than the Deny All inbound rule or the selected port.

The above-mentioned connectivity process includes two things

1. IP Range
2. My IP options

IP Range:

In this option, we can provide either a single IP or a CIDR block.

MY IP:

Case 1: If you’re connected to Azure via a public network, i.e., without any IP sec tunnel, while selecting MY IP, the IP address that is to be registered will be the Public IP address of the device you’re connecting from.

Case 2: If you’re connected to Azure via a VPN/ IP Sec tunnel/ VNET Gateway, you can’t possibly use MY IP option. Since the MY IP option directly captures the Public IP and it can’t be used. In this scenario, we need to provide the private IP address of VPN gateway for a single user, or to allow a group of users, provide private IP CIDR Block of the whole organization.

How to monitor who’s requested the access?

The users who request access are registered as Activity. To view the activities in Log analytics workspace, link the Subscription Activity to Log analytics.

It is possible to view the list of users accessed in the log analytics workspace with the help of Kusto Query Language(KQL), once it is configured to send an activity log to log analytics.

In September 2019, Azure announced a brand-new service – Azure Private Link, a very important tool for service providers providing a mix of Azure IaaS and PaaS services.

What does Azure Private Link do?

Azure Private Link enables you to access Azure PaaS Services (for example, Azure Storage and SQL Database) and Azure-hosted customer-owned/partner services over a Private Endpoint in your virtual network. Traffic between your virtual network and the service traverses over the Microsoft backbone network, eliminating exposure from the public Internet. It can be used via a local IP address (on Azure and from on-premises networks) or via a dedicated Azure ExpressRoute network.

Importance of Azure Private Link

Well, naturally, the first benefit is security!  It reduces the exposure of PaaS services to the Internet and provides a secure way to manage traffic between the client’s network and Azure. With Private Link Service, data stays within Microsoft’s system and the client’s private network.

For service providers and their clients, this is obviously critical as it provides secure access to customers in their virtual network while giving them the ability to use the resources in the service provider’s subscription.

Find out how a Private Link Service can be created behind a standard load balancer.

In the example below, Kubernetes Ingress Service is exposed as a Private Link Service. The ingress has a Standard Load Balancer with IP Address 172.17.1.100.

Details of Ingress Service (Internal Load Balancer) 

cloudiq@hubandspoke:~$ kubectl get service -A | grep  LoadBalancer
dev                ciq-demo-ingress-nginx-ingress-
controller        LoadBalancer   192.168.3.11    172.17.1.100     80:32314/TCP,443:30694/TCP   43h

Service can be accessed as below from within the VNET(ciq-demo-vnet)

http://172.17.1.100/web/api/imageresult
Added this method for testing this API in API-MGMT. The current time is : 02/20/2020 10:07:23

The private Link service is created with the following details.

• Alias – It is a unique URI identifying the service and can be accessed from anywhere within Azure.
• NAT IP – This determines the Source IP and Destination IP of incoming and outgoing packets to the Private Link service, respectively. This NAT IP can be within any subnet of the service provider VNET

Next, you create a private endpoint in the consumer vnet/subnet. In our example, we have created a network interface in the ciq-devops-general-rq-vnet/default vnet/subnet. The private ip within the vnet/subnet is 10.0.0.4. The Kubernetes ingress service can be accessed from the consumer vnet using the 10.0.0.4 private IP.

cloudiq@cloudiq-build-agent-vm:~$ curl http://10.0.0.4/web/api/imageresult
Added this method for testing this API in API-MGMT. The current time is : 02/20/2020 10:09:03

Private Link can be enabled for other Azure Resources, such as below.

For example, the private endpoint was enabled for a Storage account.

cloudiq@cloudiq-build-agent-vm:~$ curl http://k8sworkshopstg.blob.core.windows.net/test/hw.txt

Hello World!

cloudiq@cloudiq-build-agent-vm:~$ nslookup k8sworkshopstg.blob.core.windows.net
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
k8sworkshopstg.blob.core.windows.net    canonical name = k8sworkshopstg.privatelink.blob.core.windows.net.
Name:   k8sworkshopstg.privatelink.blob.core.windows.net
Address: 10.0.0.5

cloudiq@cloudiq-build-agent-vm:~$ curl http://k8sworkshopstg.privatelink.blob.core.windows.net/test/hw.txt

Hello World!

Welcome to Cloud View!

This week we look at why 2020 will be the launch of the ‘Data Decade’, top CX trends, general availability of Azure Sphere, and troubleshooting common problems in Kubernetes Deployments.

Industry News & Perspectives

Launch of the “Data Decade”

CRN asked nearly 80 CEOs five questions about how digital technologies will shape up in 2020 and beyond. Here is a summary of what they said.

Read More

Top CX Trends

Customer experience is no longer just the responsibility of client-facing departments. As more customers shop online, CX has become a top priority for CIOs as well. Here are 5 technology trends that should be a part of every CIO’s strategy.

Read More

Technology Updates

Azure Sphere

From its inception in Microsoft Research to general availability today, Azure Sphere is Microsoft’s answer to these escalating IoT threats. An interview with Galen Hunt, distinguished engineer and product leader of Azure Sphere.

DevOps and Agility

As DevOps becomes mainstream and with a range of frameworks to choose from, is DevOps losing its agility? Maybe, maybe not! Here is an article that will reconnect you to the grounding principle of DevOps – innovation, and agility.

Microsoft Datacenters in Spain

Microsoft announces its strategy for establishing new European datacenters in Spain. While there is no fixed launch date, the company has announced that the proposed DCs will deliver Azure, Microsoft 365, Dynamics 365, and the Power Platform.

From CloudIQ

Optimizing Azure Cosmos DB Performance

Azure Cosmos DB allows Azure platform users to elastically and independently scale throughput and storage across any number of Azure regions worldwide. Here is an article on how to optimize Cosmos DB performance.

Read More

How to Debug and Troubleshoot Common Problems in Kubernetes Deployments

Kubernetes deployment issues are not always easy to troubleshoot. In some cases, the errors can be resolved easily, and, in some cases, detecting errors requires us to dig deeper and run various commands to identify and resolve the issues. Here is a guided tutorial to debug applications that are deployed into Kubernetes.

Read More

Welcome to Cloud View!

This week we look at Microsoft’s Azure strategy, Gartner’s getting smarter about digital business, DevSecOps, debugging Kubernetes applications and a deep dive into Kubernetes networking.

Industry Viewpoints & News

Smarter with Gartner

Kickstart the week with this incredibly ‘smart’ article by Smarter with Gartner. The article sounds a warning about sticking to the old ways of doing digital.

Read More

Microsoft’s Azure Strategy

Gavriella Schuster, Microsoft’s Channel Chief, sat down for an illuminating chat with CRN. The conversation revealed Microsoft’s Azure strategy, its channel investment priorities, and its upcoming plans for its partners.

Read More

Google Cloud acquires Cornerstone

Google Cloud just finalized another big acquisition with Cornerstone Technology, a mainframe specialist. The new purchase fits in perfectly with Google Cloud’s strategy to make the shifting of legacy applications on to the cloud easier.

Read More

Technical Insights 

DevSecOps

When paired together, Security and DevOps can offer organizations more robust and baked-in security. Find out how companies can do DevSecOps correctly in this two-part series by Devops.com.

Azure Firewall Manager

Microsoft extends Azure Firewall Manager preview to include automatic deployment and central security policy management for Azure Firewall in hub virtual networks.

Debugging a Kubernetes Application

General troubleshooting and debugging techniques for an application running in a Kubernetes environment and the most common issues to expect.

From CloudIQ

Kubernetes Networking Deep Dive – Data Plane, how it Works Under the Hood?

Kubernetes is simple enough to get started, however one of the most complex and critical part is the networking. Here is a deep dive into the Data Plane and how it works under the hood.

Read More

Microservices – Aligning business and technology for closer collaboration, agility & flexibility

A microservices-based architecture introduces agility, flexibility and supports a sustainable DEVOPS culture ensuring closer collaboration within businesses and the news is that it’s happening for those who embrace it.

Read More

Welcome to Cloud View!

This week we look at Oracle Cloud Data Science Platform, GKE support for Windows, DevSecOps and making better quality software using Jenkins CI/CD pipeline.

Industry Perspective and News

Oracle Cloud Data Science Platform

Oracle is pulling out all stops in 2020! The company is in the news once again with the launch of its data science platform – a first of its kind cloud-native platform that is completely geared towards providing a collaborative workspace for data scientists.

Read More

Windows on GKE

Google Cloud supports Windows on GKE, as a part its commitment to providing complete support to client’s Windows server-based applications.

Read More

Six I’s of Successful IT Leaders

As every company becomes a tech company, the role of the CIOs become critical for success. Naturally, this increases the pressure on the CIOs to deliver more and think strategically. Here is an article that outlines 6 key focus areas that CIOs can start with.

Read More

Technological Insights

Storage is getting a reboot!

IBM is the first one to revamp its storage lines with the launch of Storwize and Flash Systems A9000.

DevSecOps

Security as a topic is never far off in any IT discussion. A useful webinar by CEO of Cmd, Jake King, on DevOpsTV. He lays out 7 DevOps-friendly techniques that will help you incorporate security without compromising on speed or scale.

CockroachDB

A fantastic chat with Peter Mattis, the creator of the CockroachDB open-source database and co-founder and CTO of Cockroach Labs. A conversation that covers interesting bits from his career in open source and Google.

From CloudIQ

Make Better Quality Software using Jenkins for your CI/CD Pipeline

With Jenkins, organizations can accelerate the software development process through automation. Jenkins integrates development life-cycle processes of all kinds, including build, document, test, package, stage, deploy, static analysis and much more.

Read More

Blue Green Deployment on Azure, Safe Strategy with Zero Downtime

When you are deploying a new change into production, the associated deployment should be in a predictable manner. In simple terms, this means no disruption and zero downtime!

Read More

Welcome to Cloud View!

This week we look at the State of DevOps, Hyperledger Fabric on AKS Marketplace template, Serverless computing frameworks and InfluxDB on GCP.

Industry News & Perspectives

Robot Resource Organizations

The adoption of new digital technologies and the ever-changing expectations of customers continues to challenge traditional retailers, forcing them to investigate new-human hybrid operational models, including artificial intelligence (AI), automation and robotics.

Read More

Oracle Cloud

While the cloud market pie is divided into 3-4 large slices, there is still plenty of business in the thin sliver left for ‘others’. Oracle Cloud is doing its best to capture this small market and maybe become more than a niche infrastructure provider.

Read More

State of DevOps

The 8^th annual ‘State of DevOps’ survey reports that the retail sector (as always) is the most advanced when it comes to DevOps adoption. Read on for more details.

Read More

Technical Insights

Hyperledger Fabric on AKS Marketplace template

Users with little knowledge of Azure or Hyperledger Fabric will now be able to easily set up a blockchain consortium on Azure with the new Hyperledger Fabric on Azure Kubernetes Service marketplace template.

Serverless computing frameworks

A new report by Datadog shows that almost 50% of the companies using its platform are opting for AWS Lambda serverless computing framework.

InfluxDB on GCP

Database provider InfluxDB is now live on GCP. It announced the availability of its managed cloud service as a part of Google Cloud’s open-source umbrella. Next on the agenda is the rollout of its second-generation serverless offering on Azure.

From CloudIQ

Develop Faster with Continuous Integration & the Tools to Get the Job Done

In recent years CI has become a best practice for software development and is guided by a set of key principles. Among them are revision control, build automation and automated testing.

Read More

Installing and Using HELM, the Package Manager for Kubernetes

Helm is a package manager for Kubernetes that allows developers and operators to more easily package, configure, and deploy applications and services onto Kubernetes clusters.

Read More

Welcome to Cloud View!

This week we look at Stackshare’s top 140 tools for developers in 2019, auto-labelling tool for AI developers, using Terraform for multi-cloud orchestration strategy and more.

Industry Perspective & News

Launchable aims to increase delivery velocity

This is the best use of any extra 25 minutes you have today. Jenkins founder Kohsuke Kawaguchi (KK) and respected DevOps veteran Harpreet Singh have launched a new company called Launchable that aims to increase delivery velocity. They talk all about it in this podcast (if you prefer to read, then the link has a complete transcript too).  

Read More

AWS maintains market share

January ended with the news of AWS posting revenues of $9.95 billion in Q4 2019 – bringing its 2019 revenue up to a grand total of $40bn. But, as we say, “cloud is complex,” and there is plenty of big fish in the cloud market.

Read More

Tech Insights

Top 140 tools for developers in 2019

Here is an article every developer MUST bookmark. Stackshareanalyzed over four million data points shared in their community and shortlisted the definitive list of Top 140 tools for developers in 2019!

Auto-labelling tool for AI developers

After that mammoth list, we decided to continue with the tool theme. So here is another one – a new auto-labeling tool for AI developers by IBM.

Cisco HyperFlex Application Platform (HXAP)

Cisco launched a tool (or rather a whole bouquet of tools) that lets customers build their own cloud-native environments. The HyperFlex Application Platform (HXAP) offers a whole host of integrated tools such as container networking, storage, a load balancer, and more.

From CloudIQ

Kubernetes on Azure: A 2-day workshop for AKS developers

Container technology has revolutionized the DevOps landscape and offers organizations the chance to develop and test applications faster and more cost-effectively. CloudIQ’s 2-day hands-on workshop is designed to give DevOps team members the opportunity to skill-up and learn Kubernetes design, deployment, and management.

Read More

Terraform for Multi-Cloud Orchestration Strategy

Terraform being cloud-agnostic, allows a single configuration to be used to manage multiple providers, and to even handle cross-cloud dependencies by simplifying management and orchestration.

Read More

There are many DevOps lifecycle tools out there, however GitLab is a complete package designed for coordinating CI/CD pipelines.

GitLab is a web-based DevOps lifecycle tool. This application offers functionality to automate the entire DevOps life cycle from planning to creation, build, verify, security testing, deploying, and monitoring, offering high availability and replication. It is highly scalable and can be used on-prem or on the cloud. GitLab also includes a wiki, issue-tracking, and CI/CD pipeline features.

When DevOps projects are spread across large, geographically dispersed teams a complete DevOps tool is highly useful to maintain collaboration, incorporate feedback, avoid mistakes, and speed up the development process.

GitLab goes beyond being just a repository manager; it has a built-in CI/CD, which saves enormous amounts of time and keeps the workflow smooth. Along with its own CI/CD, GitLab also allows for a range of 3^rd party integrations with external CI, so you always have the option of working with the tools based on your workflow. 

Here is a quick run-through on how to start with GitLab.

Project:

In GitLab, we can create projects for hosting codebase, use it as an issue tracker, collaborate on code, and continuously build, test, and deploy apps with built-in GitLab CI/CD. Projects can be available publicly, internally, or privately, at our choice. GitLab does not limit the number of private projects we create.

Create a project in GitLab

In the dashboard, click the green “New project” button or use the plus icon in the navigation bar. This opens the New Project page.

On the New Project page :

• Create a Blank project
• Fill the name of your project in the Project name field
• Project URL Field which is the URL path for the project that the GitLab instance will use
• Project slug field will be auto-populated
• The Project description (optional) field enables you to enter a description for the project’s dashboard
• Changing the Visibility Level modifies the project’s viewing and access rights for users
• Selecting the Initialize repository with a README option creates a README file so that the Git repository is initialized, has a default branch, and can be cloned
• Click Create project

Repository

A repository is a part of a project, which has a lot of other features.

Host your codebase in GitLab repositories by pushing files to GitLab. You can either use the user interface (UI) or connect your local computer with GitLab through the command line.

GitLab Basic Commands: https://docs.gitlab.com/ee/gitlab-basics/command-line-commands.html

Branch

When you create a new project, GitLab sets the master as the default branch for your project. You can choose another branch to be your project’s default under your project’s Settings > Repository.

Commits

When you commit your changes, you are introducing those changes to your branch. Via a command line, you can commit multiple times before pushing.

A commit message is important to identify what is being changed and, more importantly, why. In GitLab, you can add keywords to the commit message that will perform one of the actions below:

• Trigger a GitLab CI/CD pipeline: If you have your project configured with GitLab CI/CD, you will trigger a pipeline per push, not per commit.
• Skip pipelines: You can add to you commit message the keyword [ci skip], and GitLab CI will skip that pipeline.
• Cross-link issues and merge requests: Cross-linking is great to keep track of what’s somehow related in your workflow. If you mention an issue or a merge request in a commit message, they will be shown on their respective thread.

CI/CD Pipeline

Continuous Integration works by pushing small code chunks to your application’s code base hosted in a Git repository, and, to every push, run a pipeline of scripts to build, test, and validate the code changes before merging them into the main branch.

 Continuous Delivery and Deployment consist of a step further CI, deploying your application to production at every push to the default branch of the repository.

These methodologies allow you to catch bugs and errors early in the development cycle, ensuring that all the code deployed to production complies with the code standards you established for your app.

Two top-level components are:

1. .gitlab-ci.yml
2. GitLab Runner

.gitlab-ci.yml

The .gitlab-ci.yml file is where we configure what CI does with the project. It lives in the root of the repository. On any push to the repository, GitLab will look for the .gitlab-ci.yml file and start jobs on Runners according to the contents of the file, for that commit.

Pipeline configuration begins with jobs. Jobs are the most fundamental element of a .gitlab-ci.yml file.

Jobs are:

• Defined with constraints stating under what conditions they should be executed
• Top-level elements with an arbitrary name and must contain at least the script clause
• Not limited in how many can be defined

For example:

job1:
  script: “execute-script-for-job1”

job2:
  script: “execute-script-for-job2”

GitLab Runner

GitLab Runner is a build instance that is used to run jobs and send the results back to GitLab. It is designed to run on the GNU/Linux, macOS, and Windows operating systems.

Runners run the code defined in .gitlab-ci.yml. They are isolated (virtual) machines that pick-up jobs through the coordinator API of GitLab CI. If we want to use Docker, install the latest version. GitLab Runner requires a minimum of Docker v1.13.0.

Types of Runner :

1. Specific Runner – useful for jobs that have special requirements or for projects with a specific demand.
2. Shared Runner – useful for jobs that have similar requirements between multiple projects. Rather than having multiple Runners idling for many projects, you can have a single or a small number of Runners that handle multiple projects.
3. Group Runner – useful when you have multiple projects under one group and would like all projects to have access to a set of Runners. Group Runners process jobs using a FIFO (First In, First Out) queue.

How to use Runner:

1. Install Runner  –  https://docs.gitlab.com/runner/#install-gitlab-runner
2. Register Runner – https://docs.gitlab.com/runner/register/index.html

Sample Docker Project:

1. Create/Upload 2 files. .gitlab-ci.yml and Dockerfile

File Contents:
.gitlab-ci.yml:

# Official docker image.
image: docker:19.03.0-dind

services:
  - docker:19.03.0-dind

before_script:
  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY

build-master:
  stage: build
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE" .
    - docker push "$CI_REGISTRY_IMAGE"
  tags:
    - docker

Dockerfile

FROM python:2.7
RUN pip install howdoi
CMD ['howdoi']

Once we create .gitlab-ci.yml file, each push will trigger the pipeline. We didn’t create the Runner yet. So, while creating these files, add “[ci skip]” to commit message. This will skip the CI/CD pipeline.

2. Install Runner

For this example, we are using a specific runner and are going to install a runner in Windows. Refer to this link to Install Runner in Windows: https://docs.gitlab.com/runner/install/windows.html

3. Register Runner:

In order to register a runner, we need a registration token which can be found in Settings > CI/CD > Runner Tab

Check below Specific Runner section. You can copy the token from there.

To register a Runner under Windows run the following command in the path where we install GitLab Runner:
./gitlab-runner.exe register

Enter your GitLab instance URL:
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com)
https://gitlab.com

Enter the token you obtained to register the Runner:
Please enter the gitlab-ci token that we copied earlier

Enter a description for the Runner, you can change this later in GitLab’s UI:
Please enter the gitlab-ci description for this Runner
[hostname] my-runner

Enter the tags associated with the Runner, you can change this later in GitLab’s UI:
Please enter the gitlab-ci tags for this Runner (comma separated):
docker

Enter the Runner executor:
Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, Docker, parallels, virtualbox, docker-ssh, shell:
docker

If you chose Docker as your executor, you’ll be asked for the default image to be used for projects that do not define one in .gitlab-ci.yml:
Please enter the Docker image (eg. ruby:2.6):
python:2.7

Once the Runner is created successfully, it will be displayed under Settings > CI/CD > Runner > Specific Runner section.

4. Now, Go to CI/CD > Pipelines and Click  Run Pipeline Button

It will open a new window. Click Run Pipeline button again.

5. Once the job completed successfully, it will be displayed as below.

Welcome to Cloud View!

This week we look at some of the infrastructure and operations trends and insights on MLOps, Amazon Alexa and Blue green deployment on Azure.

Industry Perspective & News

AWS slashes cost of DR

Celebrate! AWS has announced massive cloud cost reductions on disaster recovery and Kubernetes services

Read More

Infrastructure & Operations Trends

Another week, another bunch of predictions. The difference here is that the article also details some trends that are on the wane.  

Read More

Tech Insights

MongoDB Supports GraphQL

MongoDB is now supporting GraphQL language for accessing its serverless application platform. This promises to extend their technology towards web and mobile apps.

CircleCI Orbs

In just a year since its launch, CircleCI Orbs are being used by over 13,000 organizations in close to 9 million CI/CD pipelines. New collaborations with 20 partners now extend the Orbs ecosystem even further.

MLOps

MLOps – the collaborative best practices that accelerate the machine lifecycle across model development, deployment, monitoring, and more – can give organizations a massive edge against competitors. John ‘JG’ Chirapurath, General Manager, Azure Data & AI explains more in this blog.

From CloudIQ

Amazon Alexa Custom Skills – How to Build One Step-by-Step

Amazon’s Alexa is the voice activated and interactive AI Bot designed to respond to number of commands and converse with people. Alexa Skills are apps that give Alexa even more abilities.

Read More

Blue Green Deployment on Azure, Safe Strategy with Zero Downtime

In Azure, different processes are available for implementing the Blue-Green strategy with two environments. In this article we discuss some of these techniques.

Read More

Cybersecurity is the number one concern for CEOs and is unanimously seen as the biggest threat in the coming years. Reports suggest that the damages from cyberattacks will to amount to $6 trillion annually by 2021.

While a lot of news coverage is given to malicious hackers and ransomware attacks, another crucial area of cyber protection is tightening the internal defenses with intelligent identity management. Keeping a tight control on who can get past your firewalls is vital for maintaining optimum security.

In this article we will review the comprehensive set of security tools available in Azure Cloud.

Azure Active Directory

Multi-Factor Authentication

Azure Multi-Factor Authentication (MFA) helps safeguard access to data and applications while maintaining simplicity for users. It provides additional security by requiring a second form of authentication and delivers strong authentication via a range of easy to use authentication methods. Users may or may not be challenged for MFA based on configuration decisions that an administrator makes.

The security of two-step verification lies in its layered approach. Compromising multiple authentication factors presents a significant challenge for attackers. Even if an attacker manages to learn the user’s password, it is useless without also having possession of the additional authentication method. It works by requiring two or more of the following authentication methods: Something you know (typically a password), Something you have (a trusted device that is not easily duplicated, like a phone), Something you are (biometrics)

Conditional Access policies

Conditional Access is the tool used by Azure Active Directory to bring signals together, to make decisions, and enforce organizational policies. Conditional Access policies at their simplest are if-then statements; if a user wants to access a resource, then they must complete an action. Example: A payroll manager wants to access the payroll application and is required to perform multi-factor authentication to access it.

Azure AD identity protection

Identity Protection is a tool that allows organizations to accomplish three key tasks: Automate the detection and remediation of identity-based risks, Investigate risks using data in the portal, Export risk detection data to third-party utilities for further analysis. The signals generated by and fed to Identity Protection, can be further fed into tools like Conditional Access to make access decisions, or fed back to a security information and event management (SIEM) tool for further investigation based on your organization’s enforced policies.

Azure AD Privileged Identity Management

Privileged Identity Management provides time-based and approval-based role activation to mitigate the risks of excessive, unnecessary, or misused access permissions on resources that you care about. Here are some of the key features of Privileged Identity Management:

• Provide just-in-time privileged access to Azure AD and Azure resources
• Assign time-bound access to resources using start and end dates
• Require approval to activate privileged roles
• Enforce multi-factor authentication to activate any role
• Use justification to understand why users activate
• Get notifications when privileged roles are activated
• Conduct access reviews to ensure users still need roles
• Download audit history for internal or external audit

Network Security

Network Security Groups (NSGs)

Network security group security rules are evaluated by priority using the 5-tuple information (source, source port, destination, destination port, and protocol) to allow or deny the traffic. A flow record is created for existing connections. Communication is allowed or denied based on the connection state of the flow record. The flow record allows a network security group to be stateful. If you specify an outbound security rule to any address over port 80, for example, it’s not necessary to specify an inbound security rule for the response to the outbound traffic. You only need to specify an inbound security rule if communication is initiated externally. The opposite is also true. If inbound traffic is allowed over a port, it’s not necessary to specify an outbound security rule to respond to traffic over the port. Existing connections may not be interrupted when you remove a security rule that enabled the flow. Traffic flows are interrupted when connections are stopped, and no traffic is flowing in either direction, for at least a few minutes.

Azure Firewall

With Azure Firewall, you can configure – Application rules that define fully qualified domain names (FQDNs) that can be accessed from a subnet and Network rules that define source address, protocol, destination port, and destination address. Network traffic is subjected to the configured firewall rules when you route your network traffic to the firewall as the subnet default gateway.

Application security groups

Application security groups enable you to configure network security as a natural extension of an application’s structure, allowing you to group virtual machines and define network security policies based on those groups. You can reuse your security policy at scale without manual maintenance of explicit IP addresses. The platform handles the complexity of explicit IP addresses and multiple rule sets, allowing you to focus on your business logic.

Resource management security

Azure resource locks

As an administrator, you may need to lock a subscription, resource group, or resource to prevent other users in your organization from accidentally deleting or modifying critical resources. You can set the lock level to CanNotDelete or ReadOnly. In the portal, the locks are called Delete and Read-only, respectively. CanNotDelete means authorized users can still read and modify a resource, but they can’t delete the resource. ReadOnly means authorized users can read a resource, but they can’t delete or update the resource. Applying this lock is similar to restricting all authorized users to the permissions granted by the Reader role.

Azure policies

Azure Policy is a service in Azure that you use to create, assign, and manage policies. These policies enforce different rules and effects over your resources, so those resources stay compliant with your corporate standards and service level agreements. Azure Policy meets this need by evaluating your resources for non-compliance with assigned policies. All data stored by Azure Policy is encrypted at rest. For example, you can have a policy to allow only a certain SKU size of virtual machines in your environment. Once this policy is implemented, new and existing resources are evaluated for compliance.

Custom RBAC roles

Granting permission using custom Azure AD roles is a two-step process that involves creating a custom role definition and then assigning it using a role assignment. A custom role definition is a collection of permissions that you add from a preset list. These permissions are the same permissions used in the built-in roles.

Once youíve created your role definition, you can assign it to a user by creating a role assignment. A role assignment grants the user the permissions in a role definition at a specified scope. This two-step process allows you to create a single role definition and assign it many times at different scopes. A scope defines the set of Azure AD resources the role member has access to.

Encryption for data at rest

Azure SQL Database Always Encrypted

Always Encrypted is a new data encryption technology in Azure SQL Database and SQL Server that helps protect sensitive data at rest on the server during movement between client and server, and while the data is in use, ensuring that sensitive data never appears as plaintext inside the database system. After you encrypt data, only client applications or app servers that have access to the keys can access plaintext data.

Implement database encryption

Transparent data encryption (TDE) helps protect Azure SQL Database, Azure SQL Managed Instance, and Azure Data Warehouse against the threat of malicious offline activity by encrypting data at rest. It performs real-time encryption and decryption of the database, associated backups, and transaction log files at rest without requiring changes to the application. By default, TDE is enabled for all newly deployed Azure SQL databases.

Implement Storage Service Encryption

Data in Azure Storage is encrypted and decrypted transparently using 256-bit AES encryption, one of the strongest block ciphers available, and is FIPS 140-2 compliant. Azure Storage encryption is similar to BitLocker encryption on Windows.

Azure Storage encryption is enabled for all new storage accounts, including both Resource Manager and classic storage accounts. Azure Storage encryption cannot be disabled. Because your data is secured by default, you don’t need to modify your code or applications to take advantage of Azure Storage encryption.

Implement disk encryption

Azure Disk Encryption helps protect and safeguard your data to meet your organizational security and compliance commitments. It uses the Bitlocker feature of Windows to provide volume encryption for the OS and data disks of Azure virtual machines (VMs), and is integrated with Azure Key Vault to help you control and manage the disk encryption keys and secrets.

Configure application security

Configure SSL/TLS certs

If you purchase an App Service Certificate from Azure, Azure manages the following tasks: Takes care of the purchase process from GoDaddy, Performs domain verification of the certificate, Maintains the certificate in Azure Key Vault, Manages certificate renewal (see Renew certificate), Synchronize the certificate automatically with the imported copies in App Service apps.

Configure and Manage Key Vault

Manage access to Key Vault

Azure Key Vault is a cloud service that safeguards encryption keys and secrets like certificates, connection strings, and passwords. Because this data is sensitive and business-critical, you need to secure access to your key vaults by allowing only authorized applications and users.

Access to a key vault is controlled through two interfaces: the management plane and the data plane. The management plane is where you manage Key Vault itself. Operations in this plane include creating and deleting key vaults, retrieving Key Vault properties, and updating access policies. The data plane is where you work with the data stored in a key vault. You can add, delete, and modify keys, secrets, and certificates.

To access a key vault in either plane, all callers (users or applications) must have proper authentication and authorization. Authentication establishes the identity of the caller. Authorization determines which operations the caller can execute.

Both planes use Azure Active Directory (Azure AD) for authentication. For authorization, the management plane uses role-based access control (RBAC), and the data plane uses a Key Vault access policy.

Welcome to Cloud View!

This week we look at the worldwide IT spending for 2020, aligning IoT and AI with business goals and the Kubernetes bug bounty program.

Tech Insights

IT Spending 2020

Gartner starts the year with some great news – Worldwide IT spending is projected to increase by 3.4% to $3.9 trillion in 2020. Get more insights from the Gartner report here.

Read More

Adoption of IoT and AI

To get the full benefits of IoT and AI, their adoption will need to merge with business strategies and goals. And that will change the way businesses are structured.

Read More

Industry Insights

Kubernetes-based Red Hat OpenShift 4.3 

Red Hat’s cloud-native commitment stays strong! The company just released its Kubernetes-based Red Hat OpenShift 4.3 and Red Hat OpenShift Container Storage 4 to provide multi-cloud Kubernetes container support.

Kubernetes bug bounty program

Kubernetes Product security committee is launching a new bug bounty program to tap into the power of the highly active Kubernetes community to find vulnerabilities in the software. Find out how you can get started.

From CloudIQ

Introduction to Machine Learning and How It Works

In this article we look at the basics of machine learning, the different algorithm models and a simple machine learning algorithm example using Python.

Read More

How to build real-time streaming data pipelines and applications using Apache Kafka?

In this article we will discuss how to use Apache Kafka, the distributed publish-subscribe messaging system and to pass messages from one end-point to another.

Read More

Welcome to Cloud View!

This week we look at some of the technologies of the future and insights on container performance & security, connected vehicles, and chatbots.

Latest in Tech

CES 2020

2020 starts with the biggest tech event of the year – CES 2020. The Las Vegas event showcases the technology of the future! Here are the highlights of some of the enterprise technologies on display.

Read More

Technology 2020

Want to know the latest technology trends that will impact businesses in 2020? From the empowered edge to human augmentation and more, here are 15 of them.

Read More

Industry Insights

Service Mesh

Containers are dominating the software world, but despite their popularity and orchestration software like Kubernetes, they are still challenging to manage. Service meshes come as the answer to improving container performance and security.

Connected Vehicles

The world of vehicle software is heating up! BlackBerry QNX and AWS are targeting automotive OEMs to bring services, personalization, health monitoring, and advanced driver assistance (ADAS) to vehicles.

Chatbots

In the next couple of years, 70% of white-collar workers will chat with conversation AI platforms daily – predicts Gartner. Here is a case study of improving customer service with an intelligent virtual assistant using IBM Watson.

From CloudIQ

Azure Database for MySQL and Grafana to monitor Azure services

More and more organizations run their business-critical applications on containers using Azure and that calls for a more intuitive dashboard to monitor and track Azure Services.

Read More

How to Create and Run Spark Clusters with Qubole using AWS

Qubole is a platform that puts big data on the cloud to power business decisions based on real-time analytics. Here is how you can create and run Spark Clusters with Qubole using AWS.

Read More

As more and more organizations run their business-critical applications on containers using Azure, there are new challenges in monitoring and managing them. Of course, there is the Azure dashboard, but with elaborate set-ups and such, IT teams feel the need for a more intuitive dashboard to monitor and track Azure services.

The answer, Grafana.

Grafana is an open-source dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus, and InfluxDB. It is a powerful visualization application that deals effectively with large-scale measurement data and time-series data.

As compared to other dashboards, especially the native Azure dashboard, Grafana offers a wider variety of visualization options (graphs, heatmaps, tables, and more) and can collect and collate data from multiple sources. It is designed for evaluating metrics such as system CPU, memory, disk, and I/O utilization.

A Grafana dashboard will help you understand, analyze, monitor, and explore your data with flexible and fast visualization tools.

In this article we will look at using Azure Database for MySQL and Grafana to monitor Azure services

Access Requirements

In your Azure subscription, your account must have “Microsoft.Authorization/*/Write” access to assign and AD app to a role. This action is granted through “Owner” role or “User Access Administrator” role. “Contributor” role will not have the required permissions.

Virtual machine requirements,

• VM Operating System      :  Linux (ubuntu 18.04)
• VM Size                                   :  Standard D2s v3 (2 vcpus, 8 GiB memory) is more than enough
• SSH access                            :  username and password.
• Default port                       :  3000
• NSG rule                                 :  open an inbound rule in network security group with
                                                      limited access to port 3000 and 22 for SSH.
• Assign a static public IP address to the VM

MySQL Creation and Linking to Grafana

1. Create an Azure database for MySQL server from

2. Select the resource group, provide Server name, admin username, password, confirm password. Take a note of the password; it is used several times throughout the set-up.

3. To select compute and storage,

a. There are three pricing tiers, (choose basic)

• Basic
• General-purpose
• Memory-optimized

b. Select the appropriate sizes.

• Computer generation: Gen 5
• vCore: 1
• storage: 5GB
• Auto-growth:
• Backup retention period:
• Local redundant / Geo-Redundant

For basic compute and storage,

The maximum vcore is 2, and Storage is 1024 GB. Choose as per your needs.

4. Then click Review+Create

5. Once the Azure database for MYSQL server is deployed, go to connection security and do the following changes,

• Add a client IP.
• Set “Allow access to Azure services” to ON

6. Do the following in the SQL server by connecting to it using the Server admin login name and password in SQL workbench. Create a new query tab. (You can use any tool to connect MySQL)

7. Run the following commands in the query tab,

• create database named “grafana” ;

8. Now the SQL server-side configurations are over. We need to provide the inputs of SQL server configuration to docker containers running Grafana.

9. Login to the VM running Grafana using appropriate SSH credentials (password or access keys).

10. Note the following values and save them as environment variables as an environment list.

Type                =          mysql
Host                =          <servername>:3306 (mysql server name created earlier)
Name              =          grafana (DB name given in the earlier steps and given access)
User                =          <Server admin login name>
Password       =          <server login password>

11. Save the changes mentioned above as a list. As shown,

Installing Grafana as a docker container and required its plugins

1. Login to the server using appropriate credentials,

2. Get updates using, sudo apt-get update

3. Install docker using the command, sudo apt install docker.io

4. Enable and start docker,

• sudo systemctl start docker
• sudo systemctl enable docker

5. Verify the installation using the command,

• docker –version the result will be like this,

6. Now login as root using the command,

• sudo su

7. Pull Grafana image; this needs an Internet connection as this will download the image from a public hub of docker

• docker pull grafana/grafana

8. Run the image with saved environment variables,

• docker run -d –name=grafana -p 8000:3000 –env-file ./env.list grafana/grafana

9. verify the container installation using “docker ps” command

10. The next step is to install the plugins for Grafana, which will be used in setting up the dashboard. We need to login to the container created previously to install these plugins.

11. Now create a shell inside the container using,

• docker exec -it grafana /bin/bash

12. The result will be as shown,

13. By default, in Grafana dashboard there’ll be limited number of panel plugins, to use more visualization we can manually install plugins. Now copy the plugin installation commands listed below and run them one by one or everything at once.

• grafana-cli plugins install michaeldmoore-annunciator-panel
• grafana-cli plugins install grafana-piechart-panel
• grafana-cli plugins install farski-blendstat-panel
• grafana-cli plugins install michaeldmoore-multistat-panel
• grafana-cli plugins install grafana-polystat-panel
• grafana-cli plugins install flant-statusmap-panel
• grafana-cli plugins install grafana-clock-panel
• grafana-cli plugins install neocat-cal-heatmap-panel
• grafana-cli plugins install briangann-gauge-panel
• grafana-cli plugins install natel-plotly-panel

14. Once the plugins are installed,

• verify the installation by going into, /var/lib/grafana/plugins directory by using the commands listed below
• cd  /var/lib/grafana/plugins
• to view installed plugins, use ls command

15. Now exit the container, command: exit

16. Now restart the container using,

• docker container restart Grafana (here “grafana” refers to the container name created earlier, which can be found using docker ps command)

Linking Azure Monitoring Tools

Service Principal

• Register an app in Azure Active Directory (AD).
• Create a client secret in the Registered app.
• Go to subscription à IAM à search for the app registration and provide “READER” access to the registered app in the Azure AD in first place.

Applying the service principal to Grafana,

• Go to Grafana UI by using public IP address followed by port number,
  i.e., (IP address):3000 example: 13.25.49.164:3000
• Now Add data source. And click select

• This page will appear, input the tenant ID, client ID, client secret.

• Then provide details for log analytics workspace and Application insights.

• If the provided details are correct this message should be displayed.

Welcome to Cloud View!

We hope you had a joyful and fun holiday! NOW with the festivities behind us, it’s time for business again!

Let’s start with what IT Leaders are planning for 2020.

Leader’s Speak

CIO’s plan for 2020

What are the CIOs thinking and planning for the coming year? It seems finding talent, dealing with rising security problems, and prioritizing the acquisition of new technologies are some of the topics occupying the C-suite.

Read More

IT Industry in 2020

Joel Friedman, CTO, Rackspace offers his take on what 2020 will hold for the IT industry. According to Joel, hybrid, and multi-cloud, SaaS and security problems will grow;

Read More

A decade since the launch of Azure

2020 marks a decade since the launch of Azure. Ever wondered what the founders think about their creation? Here’s a short interview with Microsoft’s Yousef Khalidi and Hoi Vo, key members of the original Azure ‘dream team’.

Read More

Industry insights

AI Solutions

As a digital-first service provider, we at CloudIQ help implement AI solutions across organizations. Impact like this is what we aim for! This is what the future of AI looks like.

Top Big Data companies to watch for in 2020

Big Data is going to dominate many a boardroom in the coming few years. We start the year by tracking some promising companies that will define the coming year with their next-generation data management, data science, and machine learning technology.

From CloudIQ

Deploying a Pod containing three applications using Jenkins CI/CD pipeline and updating them selectively

Kubernetes pod is a layer of abstraction wrapped around containers to group them together for resource allocation and efficient management. Here is how to deploy a pod containing three applications using Jenkins CI/CD pipeline and update them selectively.

Read More

Provisioning Cloud Infrastructure using AWS CloudFormation Templates

Spend less time managing cloud infrastructure and focus on building your application, thanks to AWS CloudFormation templates. Here is a quick start guide to creating the templates for provisioning cloud infrastructure.

Read More

A Kubernetes pod – incidentally, some say it is named after a whale pod because the docker logo is a whale – is the foundational unit of execution in a K8s ecosystem. While docker is the most common container runtime, pods are container agnostic and support other containers as well.

Simply put, a K8s pod is a layer of abstraction wrapped around containers to group them together to allocate resources and to manage them efficiently.

Continuous integration and delivery or CI/CD is a critical part of DevOps ecosystems, especially for cloud-native applications. DevOps teams frequently use Jenkins CI/CD pipelines to increase the speed and quality of collaborated software development ecosystems by adding automation. Thanks to Helm, deploying Jenkins server to K8s is quick and easy. The difficult bit is building the pipeline.

Here is a post that describes how to deploy a pod containing three applications using a Jenkins CI/CD pipeline and update them selectively.

Task on Hand:

Use a Jenkins pipeline to build a spring-boot application to generate jar file, dockerize the application to create a docker image, push the image to a docker repository and pull the image into the pod to create containers. The pod should contain 3 containers for the three applications, respectively. Upon git commit, only the container for which there is a change must be updated (rolling update).

Steps

1. Create a pipeline using groovy script to clone the respective git repo, build the project using maven, build the docker images, push it to dockerhub and pull these images to run containers in the pod.
   
2. Repeat the steps for all the three applications in separate stages. Make sure to create a separate directory in each stage to prevent conflicts when using similar files. Also, this clones the different git repos into different folders to avoid confusion.
   
3. Here is the Jenkinsfile/Pipeline script to perform the above task:

pipeline {
agent any
stages {
stage('Build1'){
    steps{
        dir('app1'){
            script{
                git 'https://github.com/cloud/simple-spring.git'
                sh 'mvn clean install'
                app = docker.build("cloud007/simple-spring")
                docker.withRegistry( "https://registry.hub.docker.com", "dockerhub" ) {
                // dockerImage.push()
                app.push("latest")
            }
        }
    }
}

}
stage('Build2'){
    steps{
        dir('app2'){
            script{
                git 'https://github.com/cloud/simple-spring-2.git'
                sh 'mvn clean install'
                app = docker.build("cloud007/simple-spring-2")
                docker.withRegistry( "https://registry.hub.docker.com", "dockerhub" ) {
                // dockerImage.push()
                app.push("latest")
            }
        }
    }
}

}
stage('Build3'){
    steps{
        dir('app3'){
            script{
                git 'https://github.com/cloud/simple-spring-3.git'
                sh 'mvn clean install'
                app = docker.build("cloud007/simple-spring-3")
                docker.withRegistry( "https://registry.hub.docker.com", "dockerhub" ) {
                // dockerImage.push()
                app.push("latest")
            }
        }
    }
}
}
stage('Orchestrate')
{
    steps{
        script{
    sh 'kubectl apply -f demo.yaml'
        }
    }
}

}
}

4. Make sure to properly configure docker and expose the dockerd in port 4243 and then change permission to allow Jenkins to use docker commands by changing permission for the /var/run/docker.sock shown.

5. Coming to integrating Kubernetes with Jenkins, it can be done using two plugins:

• Kubernetes plugin: When using this plugin, we configure the credentials to use our local cluster/azure cluster and specify the container templates for the containers to be created in the pipeline. But since all the tasks must run in containers, it is a little confusing approach. A better approach would be to use the kuberentes-cli plugin.

Refer: https://github.com/jenkinsci/kubernetes-plugin

• Kubernetes-cli plugin: It provides a withconfig() for pipeline support, which uses the configure credentials to connect to our cluster and perform kubectl commands. However when running the pipeline, the kubeconfig wasn’t  recognized for some reason, and kept giving the error ‘file not found’.

Refer: https://github.com/jenkinsci/kubernetes-cli-plugin/blob/master/README.md

Hence, we installed kubectl on the Jenkins host, configured the cluster manually, and ran shell commands from the Jenkins pipeline, where Jenkins was recognized as an anonymous user and was only granted get access but couldn’t create pods/deployments.

Here are some common problems faced during this process and the troubleshooting procedure.

• Configuring Jenkins to use local minikube cluster:
  We had trouble using both the plugins to properly configure Jenkins to create deployments as required. Using shell commands to run kubectl was also not successful since Jenkins was recognized as an anonymous user, and authorization prevented anonymous users from creating deployments.
  
• Permission for /var/run/docker.sock is reset to root after every restart, so make sure to change it to allow Jenkins to continue to use docker commands: choose Jenkins /var/run/docker.sock
  
• Installing Minikube:
  i) Started minikube cluster using hyperv as the driver and created a virtual    switch:
  minikube start –vm-driver=hyperv  –hyperv-virtual-switch=”Primary Virtual Switch”
  
  ii) Installation takes a lot of time, so we have to wait patiently, and eventually, the cluster will get configured and started. If there is a problem with apiserver, then stop the machine after SSHing into minikube vm:
  minikube ssh
  sudo poweroff
  
  iii) Then start minikube the same way.

Here are some suggested best practices

Maintaing git repo:

• Branching must be used while updating the source code or adding a particular file to the repository. Suppose you want to add a readme, then create a new branch from master, create the readme and commit it and then merge the branch with the origin.
  
• Similarly, for adding some test files/changing source code – create a new branch for testing/modification, update and commit the code and merge with the master when finished. The purpose of this is to allow easy roll back to the original master if you run into some errors when working with the new files and to prevent any conflict in code with the master.
  
• Commit only after you have tested the code properly, never commit incomplete code.
  
• Write good commit messages to keep track of the changes you have made.
  

Versioning:

• Follow the versioning convention X.Y.Z where X is incremented for a new major update/feature, Y is incremented for minor updates/minor features and Z for minor patches/bug fixes
  
• Avoid version lock that has too many dependencies in a single version. In such a scenario the package can only be updated after releasing new versions for every dependent package.
  

Docker repo:

• Use unique tags for deployment/pushing images to the repository.
  
• Always use stable tags for building images but never deploy/pull images using stable tags. Stable tags are the ones that do not roll over the updates to future versions but are bound to the current version (tag). 

Welcome to Cloud View!

The last two days of 2019 feel a bit like a waiting period – it’s a tad early to start celebrating, but it’s hard to plan anything until the new year celebrations are truly behind us and we are back at work. We think a good way to use this time would be to indulge in a bit of nostalgia and look back at how the technology landscape evolved in 2019.

Recap 2019

Kubernetes Podcast in 2019

If you deal with Kubernetes, then we are sure you follow the Kubernetes Podcast. Here is the roundup of the year’s best! Enjoy!

Read More

CRN’s 2019 Year in Review

LOVE ‘Top-10’ listicles? Here is a mammoth list of technology top 10s by CRN. From top 10 cybersecurity stories to the top 10 mobile apps of 2019 – it’s all here!

Read More

Top 10 Smarter with Gartner Articles for 2019

No report, survey, or listicle is complete without Gartner. Here are the 10 best “Smarter with Gartner articles from 2019”. 

Read More

Industry insights

What’s New with AWS

A quick video to recap the latest AWS updates and announcements (there are many!) and a web link, too, in case you want to explore categories in more detail.

IBM Z Open Editor Support for LSP

Any programmer can attest to the power – and the usefulness – of Language Server Protocol (LSP). Now its integration with IBM’s Z Open Editor opens a whole new level for coders across the globe.

From CloudIQ

Creating AWS Security Groups for Kubernetes

We are going to discuss creating security groups in AWS for Kubernetes. The goal is to set up a Kubernetes cluster on AWS EC2, having provisioned your virtual machines.

Read More

Deploy a Spring-Boot Application in Kubernetes Pod using Jenkins CI/CD Pipeline

Kubernetes has become the preferred platform of choice for container orchestration. Here is a walkthrough of how Jenkins CI/CD pipeline is used to deploy a spring boot application in K8s.

Read More

Signing off now and will see you all in the new year. Party hard and bring in 2020 in style.

Kubernetes has become the preferred platform of choice for container orchestration and deliver maximum operational efficiency. To understand how K8s works, one must understand its most basic execution unit – a pod.

Kubernetes doesn’t run containers directly, rather through a higher-level structure called a pod. A pod has an application’s container (or, in some cases, multiple containers), storage resources, a unique network IP, and options that govern how the container(s) should run.

Pods can hold multiple containers or just one. Every container in a pod will share the same resources and network. A pod is used as a replication unit in Kubernetes; hence, it is advisable to not add too many containers in one pod. Future scaling up would lead to unnecessary and expensive duplication.

To maximize the ease and speed of Kubernetes, DevOps teams like to add automation using Jenkins CI/CD pipelines. Not only does this make the entire process of building, testing, and deploying software go faster, but it also minimizes human error. Here is how Jenkins CI/CD pipeline is used to deploy a spring boot application in K8s.

TASK on Hand:

Create a Jenkins pipeline to dockerize a spring application, build docker image, push it to the dockerhub repo and then pull the image into an AKS cluster to run it in a pod.

Complete repository:

All the files required for this task are available in this repository:
https://github.com/saiachyuth5/simple-spring

Pre-Requisites:

A spring-boot application, dockerfile to containerize the application.

STEPS:

1. Install Jenkins :

• Before you install Jenkins, make sure the system has Java 8 as the default java compiler, as Jenkins does not support Java 7.
• Follow the steps given in the link: https://www.digitalocean.com/community/tutorials/how-to-install-jenkins-on-ubuntu-16-04
• Install the suggested plugins and install the Azure Container Services plugin, to connect Jenkins to your AKS cluster.

2. Connect host docker daemon to Jenkins:

• Make sure docker is running on port 4243, and expose the port 4243.  Follow this link to expose docker :
  https://success.docker.com/article/how-do-i-enable-the-remote-api-for-dockerd

• Run the command: chmod –R Jenkins:docker filename/foldername to allow Jenkins to access docker.
• Go to manage Jenkins from browser >Configure System and scroll to the bottom
• Click the dropdown ‘add cloud’ and add Docker. Add the docker host URI in the format tcp://hostip:4243
• Click verify connection to check your connection. If everything was done right, the docker version is displayed.

3. Adding global credentials:

• Go to Credentials on the Jenkins dashboard, click global credentials, and then Add credentials.
• Select the kind as Microsoft Azure Service Principal and enter the required ids, similarly save the docker credentials under type username with password.

4. Create the Jenkinsfile :

• Refer to the official Jenkins documentation for the pipeline syntax, usage of Jenkinsfile, and simple examples.
• Below is the Jenkinsfile used for this task.

Jenkinsfile:

NOTE: While this example uses actual id to login to Azure, its recommended to use credentials to avoid using exact parameters.

pipeline {
environment {
registryCredential = "docker"
}
agent any
stages {
stage(‘Build’) {
    steps{
    script {
        sh 'mvn clean install'
    }
    }
}
stage(‘Load’) {
    steps{
    script {
        app = docker.build("cloud007/simple-spring")
    }
    }
}
    stage(‘Deploy’) {
    steps{
    script {
        docker.withRegistry( "https://registry.hub.docker.com", registryCredential ) {
        // dockerImage.push()
        app.push("latest")
        }
    }
    }
}
stage('Deploy to ACS'){
    steps{
        withCredentials([azureServicePrincipal('dbb6d63b-41ab-4e71-b9ed-32b3be06eeb8')]) {
        sh 'echo "logging in" '
        sh 'az login --service-principal -u **************************** -p ********************************* -t **********************************’
        sh 'az account set -s ****************************'
        sh 'az aks get-credentials --resource-group ilink --name    mycluster'
        sh 'kubectl apply -f sample.yaml'
    }
}
}
}
}

5. Create the Jenkins project:

• Select New view>Pipeline and click ok.
• Scroll to the bottom and select the definition as Pipeline from SCM.
• Select the SCM as git and enter the git repo to be used, path to Jenkinsfile in Script path.
• Click apply, and the Jenkins project has now been created.
• Go to my views, select your view, and click on build to build your project.

6. Create and connect to Azure Kubernetes cluster:

• Create an Azure Kubernetes cluster with 1-3 nodes and add its credentials to global credentials in Jenkins.
• Install azure cli on the Jenkins host machine.
• Use shell commands in the pipeline to log in, get-credentials, and then create a pod using the required yaml file.

YAML used:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: spring-helloworld
spec:
    replicas: 1
    selector:
    matchLabels:
        app: spring-helloworld
    template:
    metadata:
        labels:
        app: spring-helloworld
    spec:
        containers:
        - name: spring-helloworld
        image: cloud007/simple-spring:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 80

Here are some common problems faced during this process and the troubleshooting procedure.

• Corrupt Jenkins exec file:
  Solved by doing an apt-purge and then apt-install Jenkins.
• Using 32-bit VM:
  Kubectl is not supported on a 32-bit machine and hence make sure the system is 64-bit.
• Installing azure cli manually makes it inaccessible for non-root users
  Manually installing azure cli placed it in the default directories, which were not accessible by non-root users and hence by Jenkins. So, it is recommended to install azure cli using apt.
• Installing minikube using local cluster instead of AKS:
  Virtual box does not support nested VTx-Vtx virtualization and hence cannot run minikube. It is recommended to enable Hyperv and use HyperV as the driver to run minikube.
• Naming the stages in the Jenkinsfile:
  Jenkins did not accept when named stage as ‘Build Docker Image’ or multiple words for some reason. Use a single word like ‘Build’, ‘Load’ etc…
• Jenkins stopped building the project when the system ran out of memory:
  Make sure the host has at least 20 GB free in the hard disk before starting the project.
• Jenkins couldn’t execute docker commands:
  Try the command usermog –a  –G docker Jenkins
• Spring app not accessible from external IP:
  Created a new service with type loadbalancer, assigned it to the pod, and the application was accessible from this new external ip.

In an upcoming article we will show you how to deploy a pod containing three applications using Jenkins ci/cd pipeline and update them selectively.

Welcome to Cloud View!

This week we present to you some fresh articles on the evolution of some of the most dominant technologies for the coming decade.

Predictions

Blockchain in 2020

2019 has been a year of great highs and some lows for the blockchain technology. The next year is going to see some significant growth and consolidation of large blockchain protocols and digital assets.

Read More

Cybersecurity in 2020

Everyone agrees that security is going to be ALL IMPORTANT in the coming year. Here is an article that positions 2020 as the year of the breach. We guarantee you will bump security to the top of your list after reading this.

Read More

Industry insights

New features in Azure Monitor Metrics Explorer

A few months ago, Microsoft’s Azure clients gave some feedback regarding the use of metrics in Azure Portal. Now the Microsoft team comes back with some new features which address the main concerns of the community.

The Update Framework (TUF)

The ninth to join the CNCF’s list of mature technologies – The Update Framework (TUF) is an open-source technology that secures software update systems.

From CloudIQ

Kubernetes Deployment Controller – An Inside Look

Kubernetes Deployment Controller helps monitor and manage the upgrade, downgrade, and scaling of services without any disruption or downtime. Here’s a detailed look at the inner workings of Kubernetes Deployment Controller.

Read More

Implementing Azure AD Pod Identity in AKS Cluster

Cloud-based identity and access management service becomes a necessity for connecting pods in AKS cluster to access other Azure cloud resources and services. Here is a detailed look at how Azure AD Pod Identity helps.

Read More