This is a summary of the book “How to stay smart in a smart world – why human intelligence still beats algorithms” written by Gerd Gigerenzer and published by MIT Press 2022.  He is a psychologist known for his work on bounded rationality and directs the Harding Center for Risk Literacy at the University of Potsdam. He is also a partner at Simply Rational – The Decision Institute.

Recent advances in artificial intelligence have juxtaposed a different form of intelligence to ours and poses a question about the role of either intelligence. With the spectrum of reactions ranging from embracing it openly to being apprehensive about its prevalence or dominance, the author picks out a cautious approach playing on the strengths and avoiding the weaknesses. With several examples and case studies, he argues that one form of intelligence works well in stable environments with well-defined rules while the other will never lose its relevance outside that world.

The salient points from this book include assertions that AI excels in stable environments and follows rules dictated by humans, AI systems don’t perform well in dynamic environments, filled with uncertainty. Humans must try out AI to get best results. In unexplored territory, simple and transparent algorithms perform better than complex ones. Among the negative impacts, ad-based model from social media platforms can be cited. It’s possible to separate human interaction with machine supervision with clear demarcation. For example, self-driving cars could be given their own dedicated lanes where possible. Market hype and profit incentives can lead companies to overcompromise and underdeliver on digital technologies.

AI wins hands down in many games such as chess, Go etc because it learns the game rules that are fixed, it is tuned by human experts and uses brute calculation to determine the best possible move. The better defined and more stable the premise, the better the performance. The flip side is self-evident with facial recognition for instance that works 99.6% of the time. In dynamic environments, the number drops significantly. When UK police scanned the faces of 170000 soccer fans in a stadium for matches with criminal database, 93% of the matches were false.

AI is good at making correlations with huge amounts of data, even some that would have escaped humans, but it cannot recognize scenarios and deal with ambiguity. For example, Maine’s divorce rate and the United States’ per capita consumption of margarine have a significant correlation but it makes no sense. Its these false findings by AI that makes them even harder to replicate leading to a lot of waste and error in areas such as health science and biotechnology and to the tune of hundreds of billions of dollars. Assertions made today such as eat blueberries to prevent memory loss, eat bananas to get higher verbal SAT score,  eat kiwis late at night to sleep better etc may just be the opposite in due time.

Whenever the effectiveness of AI decreases, human intervention can significantly boost their performance. The human brain has a remarkable ability to adapt to constantly changing cues, contexts and situations in what is termed as vicarious functioning. Staying smart means leveraging singularity capabilities but staying in charge. AI lacks four components of common sense – a capacity to think casually, an awareness of others’ intentions and feelings, a basic understanding of space, time and objects, and a longing to join in group norms. Some tasks like recommending the nearest restaurant do not need common sense but the detection of a person crossing the road in a war zone as a threat requires it.

Complex problems do not justify complex solutions. Google Flu trends tried to predict the spread of flu with approximately 160 search terms but they still overpredicted doctors’ visits. In comparison, an algorithm from Max Planck Institute for human development simply used one data point: recent visits to the doctor from the CDC website and performed much better in predicting the flu’s spread.

Information when served subliminally or unknowingly have potential to alter our behavior. This is why ad-based model for social media can be harmful by creating distractions. With attention control technology, the user is held captive by these algorithms. Texting while driving has caused 3000 deaths per year in the United States between 2010 and 2020. In areas other than driving, smartphones have proven to be very distracting.

Finally, the business aspect of artificial intelligence must be realized in the context of historical trends with killer technologies and the commerce behind it. The author says we should be able to profit from AI but not be easily misled with expectations and predictions.

Earlier book summaries: BookSummary10.docx

 

This is a continuation of articles on Infrastructure-as-code aka IaC for short.  There’s no denying that IaC can help to create and manage infrastructure and that they can be versioned, reused, and shared – all of which helps to provision resources quickly and consistently and manage them consistently throughout their lifecycle. Unlike software product code that must be general purpose and provide a strong foundation for system architecture and aspiring to be a platform for many use cases, IaC often varies a lot and must be manifested in different combinations depending on environment, purpose and scale and encompass complete development process. It can even include CI/CD platform, DevOps, and testing tools. The DevOps based approach is critical to rapid software development cycles. This makes IaC spread over in a variety of forms. The more articulated the IaC the more predictable and cleaner the deployments.  

One of the greatest advantages of using IaC is its elaborate description of the customizations to a resource type for widespread use within an organization. Among the reasons for customization, company policy enforcement, security and consistency can be called out as the main ones. For example, an Azure ML workspace might require some features to be allowed and others to be disallowed before other members of the organization can start making use of it.

There are several ways to do this. Both ARM Templates and azure cli commands come with directives to turn off features. In fact, the ‘az feature’ command line options are available on a provider-by-provider basis to register or unregister specific features. This ability is helpful separate out the experimental from the production feature set and allows them to be made available independently. Plenty of documentation around the commands and the listing of all such features makes it easy to work on one or more of them directly.

Another option is to display all the configuration corresponding to a resource once it has been selectively turned on from the portal. Since each resource type comes with its own ARM templates as well as command set, it is easy to leverage the built-in ‘show’ command to list all properties of the resource type and then edit some of those properties for a different deployment by virtue of the ‘update’ command. Even if all the properties are not listed for a resource, it is possible to find them in the documentation or by querying many instances of the same resource type.

A third option is to list the operations available on the resource and then set up role-based access control limiting some of those. This approach is favored because users can be switched between roles without affecting the resource or its deployment. It also works for groups and users can be added or removed from both groups and roles. Listing the operations enumerates the associated permissions all of which begin with the provider as the prefix. This listing is thorough and covers all aspect of working with the resources. The custom-role is described in terms of permitted ‘actions’, ‘data-actions’ and ‘not-actions’ where the first two correspond to control and data plane associated actions and the last one corresponds to deny set that takes precedence over control and data plane actions. By appropriately selecting the necessary action privileges and listing them under a specific category without the categories overlapping, we create the custom role with just the minimum number of privileges needed to complete a set of selected tasks.

Last but not the least approach, is to supply an init script with the associated resource, so that as other users start using it, the init script will set the predecided configuration that they must work with. This allows for some degree of control on sub resources and associated containers necessary for an action to complete so that by virtue of removing those resources, an action even if permitted by a role on a resource type, may not be permitted on a specific resource.

IaC is not an immutable asset once it is properly authored. It must be maintained just as any other source code asset. Part of the improvements come from fixes to defects, design changes but in the case of IaC specifically, there are other changes coming from drift detection and cloud security posture management aka CSPM.

Reference: Earlier articles on IaC shortcomings and resolutions: IacResolutionsPart35.docx

 

This is a summary of book “Chip War – The fight for world’s most critical technology” by Chris Miller and published by Scribner in 2022. He teaches international history at Tufts University’s Fletcher School of Law and Diplomacy.

Even if it is tiny, silicon chips or integrated circuits are critical to strong national economies and modern military. Its history can be traced to 1960 when US firms started making chips in Taiwan. Now that the island’s future and the possibility of conflict with China may depend on whether China reduces its reliance on imported chips and other tech. The miniaturization and fabrication of silicon chips with the help of semiconductors is a modern-day engineering marvel. It has a rich tradition of innovations and while they have been incubated in the west, they have often been produced in the east. Taiwan Semiconductor Manufacturing Company aka TSMC in Taiwan handles much of the production of chips and is a subsidized success story. Notably China still depends on the tech products designed in the Silicon Valley.

The author says China was disadvantaged, by the government’s desire not to build connections to Silicon Valley, but to break free of it. Silicon Valley remains the epicenter of the chip industry. Since 1963 when Fairchild firm began to outsource fabrication to East Asia, innovations have fueled the chip industry. Gordon Moore, the co-founder of Fairchild, predicted the growth rate in chip power in terms of the maximum number of transistors on a single computer chip and that it would double every year between 1965 to 1975. This “Moore’s Law” has proven true for more than 50 years now.

After Bob Noyce and Gordon Moore founded the semiconductor company Intel in Silicon Valley, the first product was launched with a microprocessor or a computer-on-a-chip which included both logic and memory units. Initially, the US government advocated chip production to be sent to Japan as part of redevelopment initiatives, but when Japan surpassed the chip production of US in 1986, the posture was reversed. Cheaper alternatives were provided by Samsung whose leader Lee Byung-Chul consolidated demand from Silicon Valley and made chips under their brand names.

While Silicon Valley businessman William Perry introduced microprocessors to US military through DARPA when he joined as Undersecretary for Defense, Lynn Conway and Carver Mead developed a rules-based foundation for software that automates the task of designing chips. DARPA started financing a program that helped researchers to design chips for production, giving US a military advantage and keeping Moore’s law alive. The Russians had increased espionage to copy designs from Silicon Valley, but the mass production and adoption failed and the copy-it strategy backfired handing back the technological lead to US.

TSMC had a strong tradition of being a government backed success story. Globalization of chip fabrication had not occurred, but Taiwanization had. Morris Chang left Texas Instruments to take charge of the chip industry in Taiwan and envisioned a company that would fabricate chips its customers designed which would be a “foundry” serving “fabless” chips.

John Carruthers, an Intel R&D leader realized that new lithography tools were required to drive the next wave of Moore’s law and used Extreme UV light to make chips. Intel never came up with its own EUV lithography tools but neither did its rivals Nikon and Canon from Japan. The Dutch company ASML became the sole producer of EUV lithography tools. Today, building an advanced logic fab costs twenty billion dollars, a high barrier to entry for many US and global firms. “Fabless” chip firms that design semiconductors and outsource production to TSMC and other foundries have proliferated since the late 1980s. Apple has gained more than any other company from this outsourcing trend. Today only Samsung and TSMC produce most of the sophisticated processors. The Chinese government devised a plan named Made in China 2025 with a goal of increasing domestic chip production and reducing reliance on imported chips over ten years ending in 2025. The Chinese state owns and finances many entities that appear as private equity investment firms but constitute a collective effort to “seize foreign chip firms”. In this way, China seems to be playing a bigger role in producing non-cutting-edge logic chips.

As Chinese military stepped up development and display of technologically sophisticated weapons, the US Administration banned export of US chips to Huawei, a move that devastated the company. TSMC has agreed to open a fab in US but the national security officials would prefer to match capital expenditures in all geographies by TSMC. Until that happens, the world’s dependency on Taiwan deepens.

This is a continuation of previous posts on Infrastructure-as-code aka IaC for short.  There’s no denying that IaC can help to create and manage infrastructure and that they can be versioned, reused, and shared – all of which helps to provision resources quickly and consistently and manage them consistently throughout their lifecycle. Unlike software product code that must be general purpose and provide a strong foundation for system architecture and aspiring to be a platform for many use cases, IaC often varies a lot and must be manifested in different combinations depending on environment, purpose and scale and encompass complete development process. It can even include CI/CD platform, DevOps, and testing tools. The DevOps based approach is critical to rapid software development cycles. This makes IaC spread over in a variety of forms. The more articulated the IaC the more predictable and cleaner the deployments. 

IaC is not an immutable asset once it is properly authored. It must be maintained just as any other source code asset. Part of the improvements come from fixes to defects, design changes but in the case of IaC specifically, there are other changes coming from drift detection and cloud security posture management aka CSPM. This article talks about that.

CSPM provides us with hardening guidance that helps to improve the security of the cloud deployment and is provided through the user interface for the Microsoft Defender for the cloud which is a cloud native application protection platform with a set of security measures and practices designed to protect cloud-based applications from various cyber threats and vulnerabilities efficiently and effectively. In addition to CSPM, Defender provides solutions for DevSecOps as security management at code level across cloud subscriptions and multiple pipeline environments. Defender also provides a Cloud workload protection platform aka CWPP with specific protections for servers, containers, storage, databases, and other workloads.

The CSPM capabilities are free. It includes asset discovery, continuous assessment and security recommendations for posture hardening, compliance with Microsoft Cloud Security Benchmark (MCSB) and a secure score which measures the status of the organization’s posture. Optional CSPM plan options include attack path analysis, cloud security explorer, advanced threat hunting, security governance capabilities, and tools to assess security compliance with a wide range of benchmarks, regulatory standards, and any custom security policies required by the organization, industry, or region.

Azure Policies are routinely deployed by organizations as a catch-all when IaC does not enforce consistency or when changes beyond IaC introduce security risks. CSPM allows us to define security conditions that customize a security policy. The policy translates to recommendations that identify rsource configurations that violate the security policy. Summarizing all the security postures based on the security recommendations results in a Secure score. It also provides a dashboard to see the weaknesses in the security posture.

Fixes for the recommendations that come from CSPM can work their way into the IaC at the pace that suits the organization. Care must be taken to rank the recommendations and their fixes based on priority and severity. Each recommendation is not a hard-and-fast rule and organizations may take steps to achieve the same result as what’s prescribed in the recommendation by other means. For example, many resources may be asked to turn on dedicated private networking, but organizations may find it simpler and easier to retain public networking but reduce callers to be within a set of IP ranges as called out by CIDRs. Another approach that works for organizations is to find out the set of fixes that boost the Secure score so that those fixes can be made earlier than others.

While going through all the recommendations may be exhaustive and time consuming, it is preferable to address those that are high priority or severity. There are many aspects that can determine these and a buy-in from all the stakeholders might be helpful in this regard.

 

 

This is a summary of the book “How People Learn – designing education and training that works to improve performance” by Nick Shackleton-Jones and published by Kogan Page in 2023. He is the HR Director for Talent and Learning at Deloitte, UK.

He says education is imposed on the learners often neglecting their cares and concerns. He states the problems with the schools, colleges and organizations and then offers a sensible solution. He calls for a “Affective Context Model”- a first unified learning theory which prioritizes emotions as the root of learning. He brings forward new ideas in his book and debunks many of the established ones.

He recognizes that the status quo reinforces cognition and learning which has evolved from history. There has always been a need to convert people to productive workers but the drift between what’s taught and what’s needed has widened in several contexts. Instead, the pattern of things we care about  and the world we find ourselves in are the two things that interact, shaping each other throughout our lives. If we align this learning, we find ourselves in a dream job otherwise we take up on where we are doing things that do not matter to us.

Storytelling and language have been the primary way to convey experiences and emotions to others. This learning-based system continually assesses an individual's need and liberty to explore challenges of escalating difficulty. The Affective Context Model states that we store our affective reactions to experiences and use these to reconstruct them. Learning will not always equal knowledge transfer. This stems  from the recognition that learning is a change in behavior or capability as a result of memory and memory is defined as the encoding of an affective response to an experience, which allows that experience to be reconstructed.

The deeper the personal significance, the more reliable something is learned, and we don’t often repeat our embarrassing mistakes. The brain stores what matters most. It recalls emotional moments based on our reactions. It might also encode other people’s reactions.  A loud meeting remains in our memory because of the outburst rather the point of the meeting. The author makes a point of this idea by sticking a banana under the chair of some of the attendees to his conference so that when he meets them years later, they can recall just that. He goes on to assert that there is no distinction between semantic and episodic memory.

People can share the same experiences and remember different things. This means they take-away differently. While words can express emotions and reactions just as much as objects or concepts, communication can lead to activation of similar feelings in the listener. Predicting these reactions requires a deep understanding of what is significant to the individual, highlighting the role of empathy and personalization in education. This might suggest better storytelling.

Forcing content into people’s heads is practiced widely in oriental cultures. Enlivening the lessons with enthusiasm has shown spotty results. Cramming and spewing it out during examinations and forgetting afterwards has become an academic rigor. Taking notes helps with cramming but it only helps with learning when there is an emotional connection to the material. Similarly, a group of employees making a transition to roles as first-time managers might create a seminar around the top 10 things to do and not to do as a new leader which addresses their aspirations and fears.

Personal growth is influenced by both innate characteristics and environmental influences. Discrepancies in personal values can lead to conflicts unless acknowledged and resolved. Designing learning experiences has a connection with life experiences, both involve presenting meaningful challenges and linking new concerns to pre-existing ones. A human centered approach along with a willingness to experiment and refine iteratively immensely helps. There are two types of learning intervention – pull by building resources and not courses and push by designing experiences to build capacity. Simulations boost learning impact further. When people already care about a topic, they might offer resources they can “pull” from. The best pull resources are concise

Finally, the author proposes a 5Di learning design model that provides memorable effective education. 5Di starts by defining success, discovering what matters to the learners, designing the experiences, developing the resources and tools, involving the learners and deploying the solutions in an iterative manner. The solution is either to allow the people to learn as they care, or to find a way to make them care.

 

 

Linux Kernel:

Linux Kernel is a small and special code within the core of the Linux Operating System and directly interacts with the hardware. It involves process management, process scheduling, system calls, interrupt handling, bottom halves, kernel synchronization and its techniques, memory management and process address space.

A process is the program being executed on the processor. Threads are the objects of activity within the process. Kernel schedules individual threads. Linux does not differentiate between thread and process. A multi-threaded program can have multiple processes. A process is created using the fork call. Fork call will return in the child process and in the parent process. At the time of the fork, all the resources are copied from the parent to the child. When the exec call is called, the new address space is loaded for the process.  Linux kernel maintains a doubly linked list of task structures pertaining to the processes and refers to them with process descriptors which are used to keep information regarding the processes. The size of the process structure depends on the architecture of the machine. For 32-bit machines, it is about 1.7KB. Task structure gets stored in memory using kernel stack for each process.  A process kernel stack has a low memory address and a high memory address. The stack grows from the high memory address to the low memory address and its front can be found with the stack pointer. The thread_struct and the task_struct are stored on this process address space towards the low memory address. PID helps to identify the process from one of thousands. The thread_info_struct is used to conserve memory because storing1.7KB in a 4KB process space uses up a lot. It has pointers to the task_struct structures. The pointer is a redirection to the actual data structure and uses up a very tiny space. The maximum number of processes in linux can be set in the configuration at pid_max in the nested directories of proc, sys and kernel. A current macro points to the currently executing task_struct structure. The processes can be in different process states. The first state to enter when the process is forked is the ready state. When the scheduler dispatches the task to run, it enters the running state and when the task exits, it is terminated. A task can switch between running and ready many times by going through an intermediary state called the waiting state or the interruptible state. In this state, the task sleeps on the waiting queue for a specific event. When an event occurs, and a task is woken up and placed back on the queue. This state is visible in the task_struct structure of every process. To manipulate the current process stack, there is an API to set_task_state. The process context is the context in which the kernel executes on behalf of the process. It is triggered by a system call. Current macro is not valid in the interrupt context. Init process is the first process that gets created which then forks other user space processes. The etc tab entries and init tab entries keep track of all the processes and daemons to create.  A process tree helps organize the processes. A copy on write is a technique which makes a copy of the address space when a child edits it. Until that time, all reading child processes can continue to use only one instance. The set of resources such as virtual memory, file system, and signals that can be shared are determined by the clone system call which is invoked as part of the fork system call. If the page tables need to be copied, then a vfork system call is called instead of fork. Kernel threads only run within the kernel and do not have an associated process space. Flush is an example of kernel thread. The ps -ef command lists all the kernel threads. All the tasks that were undertaken at the time of fork are reversed at the time of the process exit. The process descriptor will be removed when all the references to it are removed. A zombie process is not in the running process. A zombie process is one that is not in the running state, but its process descriptor still lingers. A process that exists before the child is a case where the child becomes parentless. The kernel provides the child with new parents.

The kernel has two major responsibilities:

-          To interact with and control the system’s hardware components.

-          To provide an environment in which the application can run.

All the low-level hardware interactions are hidden from the user mode applications. The operating system evaluates each request and interacts with the hardware component on behalf of the application.

Contrary to the expectations around subsystems, the Linux kernel is monolithic. All of the subsystems are tightly integrated to form the whole kernel. This differs from microkernel architecture where the kernel provides bare minimal functionality, and the operating system layers are performed on top of microkernels as processes. Microkernels are generally slower due to message passing between various layers. But Linux kernels support modules which allow it to be extended. A module is an object that can be linked to the kernel at runtime.

System calls are what an application uses to interact with kernel resources. They are designed to ensure security and stability. An API provides a wrapper over the system calls so that the two can vary independently. There is no relation between the two and they are provided as libraries to applications.

The /proc file system provides the user with a view of the internal kernel data structures. It is a virtual file system used to fine tune the kernel’s performance as well as the overall system.

The various aspects of memory management in Linux includes address space, physical memory, memory mapping, paging and swapping.

One of the advantages of virtual memory is that each process thinks it has all the address space it needs. The isolation enables processes to run independently of one another. The virtual memory can be much larger than physical memory in the system. The application views the address space as a flat linear address space. It is divided into two parts: the user address space and the kernel address space. The range between the two depends on the system architecture. For 32 bit, the user space is 3GB and the kernel space is 1GB. The location of the split is determined by the PAGE_OFFSET kernel configuration variable.

The physical memory is architecture-independent and can be arranged into banks, with each bank being a particular distance from the processor. Linux VM represents this arrangement as a node. Each node is divided into blocks called zones that represent ranges within memory. There are three different zones: ZONE_DMA, ZONE_NORMAL, and ZONE_HIGHMEM. Each zone has its own use with the one named normal for kernel and the one named highmem for user data.

When memory mapping occurs, the kernel has one GB address space. The DMA and NORMAL ranges are directly mapped to this address space. This leaves only 128 MB of virtual address space and used for vmalloc and kmap. With systems that allow Physical Address Extension, handling physical memories in tens of gigabytes can be hard for Linux. The kernel handles high memory on a page-by-page basis.  It maps the page into a small virtual address space (kmap) window, operates on that page and unmaps the page. The 64 bit architecture do not have this problem because their address space is huge.

The virtual memory is implemented depending on the hardware. It is divided into fixed size chunks called pages. Virtual memory references are translated into addresses in physical memory using page tables. Different architectures and page sizes are accommodated using three-level paging mechanism involving Page Global Directory, Page Middle Directory, and Page Table. This address translation provides a way to separate the virtual address space of a process from the physical address space. If an address is not in virtual memory, it generates a page fault, which is handled by the kernel.  The kernel handles the fault and brings the page into main memory even if it involves replacement.

Swapping is the moving of an entire process to and from the secondary storage when the main memory is low but is generally not preferred because context switches are expensive. Instead, paging is preferred. Linux performs swapping at page level rather than at the process level and is used to expand the process address space and to circulate pages by discarding some of the less frequently used or unused pages and bringing in new pages. Since it writes to disk, the disk I/O is slow.

Interprocess communications aka IPC occurs with the help of signals and pipes. Linux also supports System V IPC mechanisms. Signals notify events to one or more processes and can be used as a primitive way of communication and synchronization between user processes. Signals can also be used for job control.  Processes can choose to ignore most of the signals except for the well-known SIGSTOP and SIGKILL. The first causes a process to halt its execution. The second causes a process to exit. Defaults actions are associated with signals that the kernel completes. Signals are not delivered to the process until it enters running state from ready state. When a process exits a system call, the signals are then delivered. Linux is POSIX compatible so the process can specify which signals are blocked when a particular signal handling routine is called.

A pipe is a unidirectional, ordered and unstructured stream of data. Writers add data at one end and readers get it from the other end. An example is the command “ls | less” which paginates the results of the directory listing.

UNIX System V introduced IPC mechanisms in 1983 which included message queues, semaphores, and shared memory. The mechanisms all share common authentication methods and Linux supports all three. Processes access these resources by passing a unique resource identifier to the kernel via system calls.

Message queues allow one or more processes to write messages, which will be read by one or more processes. They are more versatile than pipes because the unit is a message rather than an unformatted stream of bytes and messages can be prioritized based on a type association.

Semaphores are objects that support atomic operations such as set and test. They are counters for controlled access to shared resources by multiple processes. Semaphores are most often used as locking mechanisms but must be used carefully to avoid deadlocking such as when a thread holds on to a lock and never releases it.

Shared memory is a way to communicate when that memory appears in the virtual address spaces of the participating processes. Each process that wishes to share the memory must attach to virtual memory via a system call and similarly must detach from the memory when it no longer needs the memory.

Linux has a symmetrical multiprocessing model. A multiprocessing system consists of a number of processors communicating via a bus or a network. There are two types of multiprocessing systems: loosely coupled or tightly coupled. Loosely coupled systems consists of processors that operate standalone. Each processor has its own bus, memory, and I/O subsystem, and communicates with other processes through the network medium. Tightly coupled systems consists of processors that share memory, bus, devices and sometimes cache. These can be symmetric and asymmetric. Asymmetric systems have a single master processor that controls the others. Symmetric systems are subdivided into further classes consisting of dedicated and shared cache systems.

Ideally, an SMP System with n processors would perform n times better than a uniprocessor system but in reality, no SMP is 100% scalable.

SMP systems use locks where multiple processors execute multiple threads at the same time. Locking must be limited to the smallest time possible. Another common technique is to use finer grain locking so that instead of locking a table, only a few rows are locked at a time. Linux 2.6 removes most of the global locks and locking primitives are optimized for low overheads.

Multiprocessors demonstrate cache coherency problem because each processor has an individual cache, and multiple copies of certain data exist in the system which can get out of sync.

Processor affinity improves system performance because the data and the resources accessed by the code will stay local to the processor’s cache due to warmth. Affinity helps to use these rather than fetch repeatedly. Use of processor affinity is accentuated in Non-uniform Memory Access architectures where some resources can be closer to a processor than others.

Linux supports several file systems. The Virtual File System Interface allows Linux to support many file systems via a common interface. It is designed to allow access to files as fast and efficiently as possible.

Ex2fs was the original file system, and it became widely popular allowing typical file operations such as to create, update, and delete files, directories, hard links, soft links, device special files, sockets, and pipes. It suffered from one limitation that if the system crashed, the entire file system would be validated and corrected for inconsistencies before it is remounted. This was improved with journaling where every file system operation is logged before the operation is executed and the log is replayed to bring the file system to consistency.

Linux Volume Managers and Redundant Array of Inexpensive Disks (RAID) provide a logical abstraction of a computer’s physical storage devices and can combine several disks into a single logical unit to provide increased total storage space as well as data redundancy. Even on a single disk, they can divide the space into multiple logical units, each for a different purpose.

Linux provides four different RAID levels. RAID-Linear which is a simple concatenation of disks that comprise the volume. Raid-0 is a simple striping where the data that is written is interleaved in equal-sized “chunks” across all disks in the volume. RAID-1 is mirroring where  all data is replicated on all disks in the volume. A RAID-1 volume created from n disks can survive the failure of n-1 of those disks. RAID-5 is striping with parity which is similar to RAID-0 but with one chunk in each stripe containing parity information instead of data. RAID-5 can survive the failure of any single disk in the volume.

A Volume-Group could be used to form a collection of disks also called Physical-Volumes. The storage space provided by these disks is then used to create Logical-Volumes. It is also resizable.  New volumes are easy to add as extents and the Logical Volumes can be expanded or shrinked and the data on the LVs can be moved around within the same Volume-Group.

Beyond the hard disk, keyboard and console that a Linux system supports by default, a user-level application can create device special files to access other hardware devices. They are mounted as device nodes in the /dev directory. Usually, these are of two types: a block device and a character device. Block devices allow block-level access to the data residing on a device and the character devices allow character-level access to the devices. The ls -l command will show a ‘b’ for block device and a ‘c’ for character device in the permission string. The virtual file system devfs is an alternative to these special devices. It reduces the system administrative task of creating device node for each device.  A system administrator can mount the devfs file system many times at different mount points but changes to a device node is reflected on all the mount points. The devfs namespace exists in the kernel even before it is mounted which makes the device node, to become available independently of the root file system.

Linux also supports FUSE which is a user-space file-system framework. It consists of a kernel module (fuse.ko), a userspace library(libfuse.*) and a mount utility (fusermount). One of the most important features of FUSE is allowing secure non-privileged mounts. One example of this is the sshfs which is a secure network filesystem using the sftp protocol.

 This is a continuation of articles on Infrastructure-as-code aka IaC for short.  There’s no denying that IaC can help to create and manage infrastructure and that they can be versioned, reused and shared – all of which helps to provision resources quickly and consistently and manage them consistently throughout their lifecycle. Unlike software product code that must be general purpose and provide a strong foundation for system architecture and aspiring to be a platform for many use cases, IaC often varies a lot and must be manifested in different combinations depending on environment, purpose and scale and encompass complete development process. It can even include CI/CD platform, DevOps, and testing tools. The DevOps based approach is critical to rapid software development cycles. This makes IaC spread over in a variety of forms. The more articulated the IaC the more predictable and cleaner the deployments.

The IAC architecture is almost always dominated by the choice of technology stacks. There is no universal system architecture but a more devops oriented tailored approach with all the tools necessary to keep the deployments consistent and repeatable. Technology varies with cloud native forms, providers like Ansible, Terraform, and domain specific language such as Pulumi. IaC can be curated as a set of machine-readable files, descriptive model, configuration template, and an imperative approach.  Then there are two approaches for writing it which are an imperative approach and a declarative approach. The imperative approach allows users to specify the exact steps to be taken for a change and the system does not deviate from them while a declarative approach specifies the final form and the tool or platform involved goes through the motion of provisioning them.

Infrastructure can be made available as a service and shared as code.  Provisioning infrastructure can be a cloud service and many public clouds offer it in their service portfolio. These so-called native infrastructures are great for leveraging the public cloud built-in features but more than usual, organizations build a veritable library of assets and prefer it to not be limited to any one cloud-based resources. It can even include on-premises infrastructure. No matter what choices are made and the decision process for navigating the IaC landscape, it is unquestionable that IaC reduces shadow IT within organizations, integrates directly with CI/CD platforms, version controlling infrastructure and configuration changes, standardizing infrastructure, effectively managing configuration drift and with the ability to scale up or out without increasing CapEx or OpEx.

Configuration Management is separate from infrastructure management although tools like Ansible provide hybrid solutions. True configuration management is demonstrated by software like CFEngine while infrastructure management is demonstrated by providers like Terraform and Pulumi. Businesses can mix and match any tool and use them in their CI/CD pipelines depending on their custom requirements.

As a real-world example, a developer writes application code and the configuration management related instructions that will trigger actions from the virtualization environment. When the code is delivered, the configuration management and infrastructure management provide a live operational environment for testing. When the tests run and the error detection and resolution occur, the new code changes become ready for deployment to customer facing environments. Managing the state drift as changes keep propagating is one of the core management routines for Infrastructure-as-code.

#codingexercise

Q: An array A of N elements has each element within the range 0 to N-1. Find the smallest element P such that every value that occurs in A also occurs in sequence A[0], A[1] ... A[P]

For example, A = [2,2,1,0,1] and the smallest value of P is 3 where elements 2,2,1,0 contain all values that occur in A.

A: 

public int getPrefix(int[] A) {

Int prefix = Integer.MIN_VALUE;

Int n = A.length;

Int visited = new int[n];

for (int i = 0; i < n; i++) {

     if (visited[A[I]] == 0){

         visited[A[I]] = 1;

         prefix = I;

     }

}

return prefix;

}