Programmers Quickie: Recent Episodes
Software Engineering
Software Engineering Best Practices, System Design, High Scale, Algorithms, Math, Programming Languages, Statistics, Machine Learning, Databases, Front Ends, Frameworks, Low Level Machine Structure, Papers and Computing, Computer Science Book Reviews - Everything!
📕BOOK - Set Boundaries, Find Peace - https://amzn.to/48krxnn
AWS networking provides a broad set of services that enable you to , connect, and secure your applications and data. With AWS networking, you can easily connect your on-premises networks to AWS, create virtual private clouds (VPCs), and use a variety of networking services to route traffic, control access, and monitor your network.
- Scalability: AWS networking is designed to scale with your needs. You can easily add or remove resources as your traffic demands change.
- Reliability: AWS networking is built on a highly reliable global infrastructure. Your applications will be available even if there is a problem with one of your network components.
- Security: AWS networking provides a variety of security features to protect your applications and data. You can control who has access to your network and what they can do.
AWS networking involves creating, configuring, and managing networks on Amazon Web Services. It includes services like Virtual Private Cloud (VPC), Elastic Load Balancing (ELB), Route 53, and Direct Connect, allowing users to build secure, scalable, and highly available cloud infrastructures. Key concepts include subnets, security groups, network access control lists (ACLs), and gateways.
Programming interview patterns involve understanding data structures, algorithms, time/space complexity, problem-solving approaches, and coding practices. Familiarize yourself with arrays, strings, linked lists, trees, graphs, dynamic programming, recursion, backtracking, bit manipulation, and system design. Practice coding, optimize solutions, and enhance communication skills to efficiently express problem-solving processes during technical interviews.
rollup.200 word summary for sql cube and rollupSQL (Structured Query Language) is a programming language that is commonly used to manage and manipulate data in relational databases. Two common operations that can be performed on data in SQL are cube and rollup.A SQL cube operation calculates aggregate values across multiple dimensions, producing a multi-dimensional result set. The result set includes all possible combinations of values across the specified dimensions. For example, if we have data on sales by region, product, and date, a cube operation could be used to calculate total sales by region and product for each date. This would produce a result set with three dimensions: region, product, and date.A SQL rollup operation is similar to a cube operation, but it only calculates aggregate values for a subset of the dimensions. Specifically, a rollup operation calculates aggregate values for each combination of values in a specified subset of dimensions, and then for each level of aggregation, it calculates a total for the remaining dimensions. For example, if we have data on sales by region, product, and date, a rollup operation could be used to calculate total sales by region for each date, and then a grand total for all dates. This would produce a result set with two dimensions: region and date.Both cube and rollup operations are useful for summarizing large amounts of data and analyzing it across multiple dimensions. They can be used to generate reports and visualizations that help users better understand their data and make informed decisions based on it.
Acid transactions are a set of properties that ensure reliable and consistent database transactions. The term "ACID" stands for Atomicity, Consistency, Isolation, and Durability.
In this episode of programmers quickie podcast, we continue our exploration of SQL pivot and dive deeper into the syntax behind this powerful data analysis tool. If you're looking to take your pivot table skills to the next level, this episode is for you!
In a system design interview, you will be asked to design a complex system, such as a website, mobile app, or network infrastructure. The interviewer will evaluate your ability to decompose the problem into smaller subproblems, make trade-offs between different design decisions, and consider the scalability and performance of the final solution. The key to success in a system design interview is to clearly communicate your thought process and trade-offs, rather than getting bogged down in low-level implementation details.
In Apache Spark, you can cache a DataFrame in memory using the cache() or persist() method. The cache() method is a shorthand for persist() with the default storage level of MEMORY_ONLY. You can use the persist() method to specify a different storage level if desired, such as MEMORY_AND_DISK or MEMORY_ONLY_SER.
The n-queens problem is a well-known problem in computer science and mathematics that involves placing n chess queens on a n x n chessboard such that no two queens are able to attack each other. This means that no two queens can be placed on the same row, column, or diagonal. The problem can be generalized to other chess pieces, such as the n-rooks problem (where the goal is to place n rooks on the board such that no two rooks can attack each other) or the n-knights problem (where the goal is to place n knights on the board such that no two knights can attack each other).The n-queens problem is often used as a test case for algorithms that solve combinatorial search problems, as it is a challenging problem that requires careful consideration of the constraints and the search space. There are several ways to solve the n-queens problem, including backtracking algorithms, constraint satisfaction algorithms, and heuristic search algorithms.In LeetCode, the n-queens problem is represented as problem 51 and can be found at the following link: https://leetcode.com/problems/n-queens/. The problem asks the user to implement a function that takes in an integer n and returns a list of all valid configurations of n queens on an n x n chessboard. The function should return an empty list if no valid configurations exist.
Consistent Hashing is a distributed hashing scheme that operates independently of the number of servers or objects in a distributed hash table by assigning them a position on an abstract circle, or hash ring. This allows servers and objects to scale without affecting the overall system.
Here are scenarios where we would want to use AA testing before trying out and AB testing
HTTP/1 is a textual data protocol and is good in terms of readability, ease of reimplementing, and ease of debugging. However, Binary protocols(Used in HTTP/2 )
In probability theory and statistics, Bayes' theorem, named after Thomas Bayes, describes the probability of an event, based on prior knowledge of conditions that might be related to the event.
The token bucket is an algorithm used in packet-switched and telecommunications networks. It can be used to check that data transmissions, in the form of packets, conform to defined limits on bandwidth and burstiness.
In software design and engineering, the observer pattern is a software design pattern in which an object, named the subject, maintains a list of its dependents, called observers, and notifies them automatically of any state changes, usually by calling one of their methods.
There are six (6) criteria that should be applied when evaluating any Web site: authority, accuracy, objectivity, currency, coverage, and appearance.
The dataclass() decorator examines the class to find fields. A field is defined as a class variable that has a type annotation. With two exceptions described below, nothing in dataclass() examines the type specified in the variable annotation.
Python dependency management and packaging made easy. ... Poetry comes with all the tools you might need to manage your projects in a deterministic way
In computer networks, rate limiting is used to control the rate of requests sent or received by a network interface controller. It can be used to prevent DoS attacks and limit web scraping
dbt - Transform data in your warehouse What is dbt? dbt is a development framework that combines modular SQL with software engineering best practices to make data transformation reliable, fast,
HashAggregateExec is a unary physical operator (i.e. with one child physical operator) for hash-based aggregation that is created (indirectly through AggUtils.
The Spark SQL shuffle is a mechanism for redistributing or re-partitioning data so that the data is grouped differently across partitions
We have two types of automatic backups in dynamodb one is point in time and another one is snapshots.
Giorgia Meloni is an Italian politician and journalist. A member of the Chamber of Deputies in Italy since 2006, she has led the Brothers of Italy political party since 2014, and has been the president of the European Conservatives and Reformists Party since 2020.
📚 Book - The Psychology of Money - https://www.amazon.com/Psychology-Money-Timeless-lessons-happiness/dp/0857197681#?&_encoding=UTF8&tag=planetizer0c-20&linkCode=ur2&linkId=c18c78f3d241db79ce045de652b93722&camp=1789&creative=9325
Why a single file on your repository that describes the project could make life much easier for newcomer programmers
DataFrame.createOrReplaceTempView - Creates or replaces a local temporary view with this DataFrame.
The lifetime of this temporary table is tied to the SparkSession that was used to create this DataFrame.
Spark SQL is a Spark module for structured data processing. It provides a programming abstraction called DataFrames and can also act as a distributed SQL query engine. It enables unmodified Hadoop Hive queries to run up to 100x faster on existing deployments and data.
Apache Spark Paired RDDs are defined as RDDs containing the key-value pair(KVP), which consists of two linked data item in it. So, In most cases, the key is an identifier, and the value is data corresponding to the key value. Furthermore, Apache Spark operations work on RDDs that contain any objects.
What is this name delta lake? Why delta? What is the core benefit of using it over a standard data lake? What are its disadvantages
In this episode we will discuss what are Go language to link and how is it different from other computer languages
Java fork join pool uses the stealing algorithm in order to utilize better the threads that we have
In AWS the write capacity units is one write per second for an item of up to 1 KB in size
Why does reserve currency metals so much today in today's economy and what are the alternatives and investment opportunities in this area.
Common table expressions are very useful in constructing SQL and are your great next milestone to using SQL
What is the core of doing SEO did it change over the years and what would bring you to the topmost search results in the search engines
Here we discuss some of the most interesting product interview questions in the data area and in the data world
What does term frequency means and what is its relationship with inverse document frequency that we use in order to identify categories of documents and in order to find testament and many more applications of this topic
📚 Book - https://amzn.to/39ty0Tq - "Terraform: Up & Running: Writing Infrastructure as Code"
- Terraform
*init*will download all the dependencies needed by the terraform project like when you have a clean maven project, and running almost any mvn command will pull down the dependencies. tagsin terraform helps you set the additional tag properties of the resources, if you want to set the name of the resource that would appear in aws then you need to set the tag with specific key and this key isNameactual names of resources, so if you would like to create a vpc resource in aws with name "myvpc" you need to actually set its tags which is different from aws tags you would set thetags { Name = "MyVpc" }and this would set the name of the resource in aws and not the tags in aws.- Create EC2The terraform name
aws_instanceThe 2 mandatory arguments areamiandinstance_typeExample
``` resource "aws_instance "MyEc2Instance" {
ami = "ami_287428346" // some aws ami
instance_type = "t2.micro"
}
```
- Note to reference an id of a terraform instance need to specify
. .id so aws_vpc.myvpc.id - When we set an "aws_eip" which is elastic ip we have to tell it for which ec2 instance we assign it to, so the resource contains the "instance" property that we have to set.
- Aws account can give us up to 5 elastic ips.
- Security group - statefull firewall - stateful means if we allowed some traffic going in it would automatically allow traffic to go out, in stateless we have to set explicitly the input to the firewall and the output.
``` resource "aws_security_group" "MySecurityGroup" {
name = "Allow HTTPS"
ingress {
from_port = 443
to_port = 443 # This is a range of ports from to to is not output.
protocol = "TCP"
cird_blocks = ["0.0.0.0/0"] # All traffic.
}
egress {
}
}
```
Then we can reference this MySecurityGroup from various resources such as ec2 instances.
- Dynamic Blocks - iterator, for_each var.mylistvar , content = port.value This allows us to use for_each.prefix the block you want to use the list for each in dynamic example
``` variable "ingressrules" { # Here you set the list variable holding the data to be dynamically included.
type = list(number)
default = [80, 443]
}
dynamic "ingress" { # Now iterate and use the above list defined above.
iterator = port
for_each = var.ingressrules
content {
from_port = port.value
}
}
```
Terraform dynamic blocks and security groups definition in Terraform and AWS elastic IP definition in Terraform and EC2 We continue also to investigate what does terraform init means.
AWS elastic IP is not dynamic for itself it's actually a static public IPv4 IP so why is it called elastic because we dynamically assign it to host so if our host or easy to instance is down we can programmatically and dynamically assign such a static public IP into this new host so it's used for resiliency
With terraform we can use variables for example an integer which is defined as a number or a string or a list of strings and then we can reference these variables further down in the code in order to reuse them you can also output variables to the screen and we could also ask the customer to input interactively variable values into the console
``` In this session
- init plan apply destroy
- state file
-
variables
-
terraform pattern consistent
- resource keyword
- resource identifier "aws_vpc"
- our name "mypc"
-
in curly braces the parametesrs like cidr_block = "10.0.0.0/16"
-
terraform init - i's like mvn install
- Download the plugins to use aws
- We can run this command after created main.tf
-
pull down modules
-
terraform plan - will show what's its going to do
-
checks state
- Get developer style diff before commit
- What already created which diff
- don't want to loose this state file
-
terraform apply - showing the diff
- click
yesto continue -
resources created in aws!
-
terraform destroy - to delete all reources
-
confirm
-
state fileterraform.tfstate - we can see locally all that we created with terraform!
- this file will have ALL resources created by terraform
- visibility!
- we want this file to get missing
- json format
- .bak
- track changes we made
- used for diff
Variables
- define variables
- folder
variables defaultis like the value- we have
typeand thendefaultfor default value -
main.tfcreate file in that folder ```json variable "vpcname" { type = string (or number boolean list(string) with values ["val1", "val2"] to access 1 [0]) default = "myvpc" }variable "mymap" { type = map default = { key1 = "value1" key2 = "value2" } }
variable "mytuple" { type tuple([string, number, string]) default = ["mystr1", 45, "mystr2"] }
variable "myobject" { type = object({name = string, port = number}) default { name = "myval1" default = "232" } } ``` - cidr_range would get used multiple times so just put it in var
-
use variables
var.myvarnamecan also be done with string interpolationvar.mylist[0]-
var.mymap["Key1"] -
interactive input variables
json variable "inputname" { type = string description = "please enter value for input name" }
next use this
- var.inputname as usuall
- terraform plan will now ask to input the input vars values
Outputs
json
output "myoutput" {
value = aws\_vpc.myvpc.id
}
so nw we can print the dynamic id we get when we run terraform.
run terraform plan will not print the id as it's created only after the apply is run and resources actually run.
```
Terraform main operations such as init plan apply and destroy and the state file telephone is a infrastructure as code tool a command line to help us create infrastructure with standard adjacent files and then run telephone apply in order to actually create this infrastructure either in AWS cloud or Google cloud or azure cloud or any other cloud provider even locally and it has many plugins.
In this session
- init plan apply destroy
- state file
- variables
- terraform pattern consistentresource keyword
resource identifier "aws_vpc"
our name "mypc"
in curly braces the parametesrs like cidr_block = "10.0.0.0/16"
* terraform initi's like mvn install
Download the plugins to use aws
We can run this command after created main.tf
pull down modules
* terraform planwill show what's its going to do
checks stateGet developer style diff before commit
What already created which diff
don't want to loose this state file
* terraform applyshowing the diff
click yes to continue
resources created in aws!
* terraform destroyto delete all reources
confirm
* state file terraform.tfstatewe can see locally all that we created with terraform!
this file will have ALL resources created by terraform
visibility!
we want this file to get missing
json format
.bak
track changes we made
used for diff
``` Terraform main operations and components are:
-
[1] init [2] plan [3] apply [4] destroy
-
State files
-
variables
-
terraform pattern consistent
-
resource keyword
-
resource identifier "aws_vpc"
-
our name "mypc"
-
in curly braces the parametesrs like cidr_block = "10.0.0.0/16"
-
terraform init -
i's like mvn install
-
Download the plugins to use aws
-
We can run this command after created main.tf
-
pull down modules
-
terraform plan -
will show what's its going to do
-
checks state
-
Get developer style diff before commit
-
What already created which diff
-
don't want to loose this state file
-
-
terraform apply -
showing the diff
-
click
yesto continue -
resources created in aws!
-
terraform destroy -
to delete all reources
-
confirm
-
state fileterraform.tfstate -
we can see locally all that we created with terraform!
-
this file will have ALL resources created by terraform
-
visibility!
-
we want this file to get missing
-
json format
-
.bak
-
track changes we made
-
used for diff
```
Telephone is a language declarative language for defining resources here we will describe their form in it and the command line how to install with regards to interjeed idea plugin and also the visual studio plugin
What are the differences in between the Bernoulli the binomial and the poison distributions and what are they
Dynamo the big global tables allow us to have application of course region of a multi master multi replica database
This is going to be how to create a model with pandas very quickly in a few lines a linear regression and random voice and then to fit it to predict the output to separate in between x and y
The US economy currently has a 30 trillion depth and is trying to fight inflation by rising interest rates and also by doing quantitative tightening
In machine learning versus statistics and statistics we are trying more to describe things and in machine learning to predict we have supervised and unsupervised learning and supervised is divided further into classification and regression
The Apache Airflow project is an orchestration project for learning jobs offline managing them it has a web UI it has a scheduler.
He loves some tips about the go programming language such as a go channels go routines how to serialize and deserialize Jsons how to create tests how to want test coverage and more.
Everyone says rebalance but it takes time right? Here is my tip on how to make it super easy!!!
The Sorrows of Young Werther is a 1774 epistolary novel by Johann Wolfgang von Goethe, which appeared as a revised edition in 1787. It was one of the main novels in the Sturm und Drang period in German literature, and influenced the later Romantic movement
pbcopy will allow you to copy the output of a command right into your clipboard. Vice-versa for pbpaste — it will allow you to paste your clipboard
In computer science, a segment tree, also known as a statistic tree, is a tree data structure used for storing information about intervals, or segments. It allows querying which of the stored segments contain a given point
In computer science, a topological sort or topological ordering of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering. ... Topological sorting has many applications especially in ranking problems such as feedback arc set
The PMF is one way to describe the distribution of a discrete random variable. ... The cumulative distribution function (CDF) of random variable X is defined as FX(x)=P(X≤x), for all x∈R. Note that the subscript X indicates that this is the CDF of the random variable X. Also, note that the CDF is defined for all x∈R.
Do you document the architecture? How? How do we document it while keeping it from stale? Are the patterns? And what should we write there to help programmers the most?
The principal components of a collection of points in a real coordinate space are a sequence of p unit vectors, where the i-th vector is the direction of a line that best fits the data while being orthogonal to the first i-1 vectors
Zero is a web framework to simplify modern web development. It allows you to build your application without worrying about package management or routing.
Jinja
Jinja

Jinja is a fast, expressive, extensible templating engine. Special placeholders in the template allow writing code similar to Python syntax. Then the template is passed data to render the final document.
FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.
The key features are:
- Fast: Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic). One of the fastest Python frameworks available.
- Fast to code: Increase the speed to develop features by about 200% to 300%. *
- Fewer bugs: Reduce about 40% of human (developer) induced errors. *
- Intuitive: Great editor support. Completion everywhere. Less time debugging.
- Easy: Designed to be easy to use and learn. Less time reading docs.
- Short: Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs.
- Robust: Get production-ready code. With automatic interactive documentation.
- Standards-based: Based on (and fully compatible with) the open standards for APIs: OpenAPI (previously known as Swagger) and JSON Schema.
GraphQL is an open-source data query and manipulation language for APIs, and a runtime for fulfilling queries with existing data.
Keynesian economics are various macroeconomic scenarios about how economic output is strongly influenced by aggregate demand. In the Keynesian view, aggregate demand does not necessarily equal the productive capacity of the economy. Instead, it is influenced by a host of factors
htmx allows you to access AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext
GameStop should be understood in the whole context of the current stock market the current social media the current internet and the current investors
I just wanted to see how does the code looks like as someone who does not use neither what would appear more convenient in my eyes
Rapidly build modern websites without ever leaving your HTML.
A utility-first CSS framework packed with classes like flex, pt-4, text-center and rotate-90 that can be composed to build any design, directly in your markup.
Io Nio blocking io non blocking io nodejs java thread processed let's do some cleanup in the mess.
Agile, GTD getting things done , deep work , zero inbox, being more efficient , scrum, sprints
In mathematics, an integral assigns numbers to functions in a way that can describe displacement, area, volume, and other concepts that arise by combining infinitesimal data. Integration is one of the two main operations of calculus; its inverse operation, differentiation, is the other.
Snowflake uses s3 in order to store the data and it has a layout on top of it in order to run all the computations that you need such as SQL in this episode we are going to scan quickly the architectural so do you get what snowflake is
We chat is actually in operation system that you can install sub applications be aware of the world with the GPS look outside with the camera pay get coupons interact with the people and set up appointments
The steps and the stages you can take in order to create clear and clean SQL and comparing it to standard programming languages
DNS is just another high scale service we can learn from how it was designed for future high scale services architecture
The floating point number representation in computers is another data type but when should you be using or not, what are its advantages and limitations
Editors command line intellij visual studio vim bash zsh markdown wiki and looking what are the best practice tools for a programmer today
Here are 10 tips for productivity specifically for programmers from the top brilliant minds of productivity super experts
The differences between the basic sorting algorithms selection bubble and insertion there are delicate ones.
How do companies deal and how should they deal with detention between efficiency wounds and innovation do this collide and what can we do about it What do companies actually do about this tension. And the Netflix example from the recent Netflix CEO book.
BFS breadth first search is another building block algorithm. It's built from a queue and a visited marker array.
We all use java.util.Random let's see what a professor of mathematics who recently won $3m prize the largest one in math has to say about randomness.
Building more complex graph algorithms based on the basic building blocks such as DFS and from there moving on to connected components shortest path traveling salesman
What are the smells for a bad design document, What makes a design document good, what should you include in it, what should you not include in it guidelines on writing top notch design documents for software engineers.
Machine learning offers a fantastically powerful toolkit for building useful complexprediction systems quickly. This paper argues it is dangerous to think ofthese quick wins as coming for free. Using the software engineering frameworkof technical debt, we find it is common to incur massive ongoing maintenancecosts in real-world ML systems. We explore several ML-specific risk factors toaccount for in system design. These include boundary erosion, entanglement,hidden feedback loops, undeclared consumers, data dependencies, configurationissues, changes in the external world, and a variety of system-level anti-patterns.
In statistical hypothesis testing, a result has statistical significance when it is very unlikely to have occurred given the null hypothesis.
In computer science, a B-tree is a self-balancing tree data structure that maintains sorted data and allows searches, sequential access, insertions, and deletions in logarithmic time. The B-tree generalizes the binary search tree, allowing for nodes with more than two children.[2] Unlike other self-balancing binary search trees, the B-tree is well suited for storage systems that read and write relatively large blocks of data, such as discs. It is commonly used in databases and file systems.
Peter Naur (25 October 1928 – 3 January 2016)[1] was a Danish computer science pioneer and Turing award winner. He is best known as a contributor, with John Backus, to the Backus–Naur form (BNF) notation used in describing the syntax for most programming languages. He also contributed to creating the language ALGOL 60.
Fast development. Flutter's hot reload helps you quickly and easily experiment, build UIs, add features, and fix bugs faster. Experience sub-second reload times..
Vim is an extremely powerful editor. Not only you can use it from command line but also from IntelliJ, Visual Studio Code, and for actual command line manipulation.
ISO 8601 Data elements and interchange formats – Information interchange – Representation of dates and times is an international standard covering the exchange of date- and time-related data. It was issued by the International Organization for Standardization and was first published in 198
Should you invest time in using Vim for software development? What should you do with Vim? How do you search for files? What about syntax coloring? Loading files from history, autocomplete and more. As it turns out you don't need plugins for all this.
Why Google Stores Billions of lines of code in a single repository - this is a brief summary of a research paper published by google.
In a paper named Fixed it for you. Protocol repair using lineage graphs it is described how machine can autocorrect programming error.
The Bellman–Ford algorithm is an algorithm that computes shortest paths from a single source vertex to all of the other vertices in a weighted digraph.
Dijkstra's algorithm is an algorithm for finding the shortest paths between nodes in a graph, which may represent, for example, road networks. It was conceived by computer scientist Edsger W. Dijkstra in 1956 and published three years later. The algorithm exists in many variants
As developers, we sometimes want to check the performance of our server, we are not system administrators so we do not run these commands on a daily basis but here is a quick overview of it.
In computer science, selection sort is an in-place comparison sorting algorithm. It has an O(n²) time complexity, which makes it inefficient on large lists, and generally performs worse than the similar insertion sort
cgroups is a Linux kernel feature that limits, accounts for, and isolates the resource usage of a collection of processes. Engineers at Google started the work on this feature in 2006 under the name "process containers".
Backtracking is a general algorithm for finding all solutions to some computational problems, notably constraint satisfaction problems, that incrementally builds candidates to the solutions, and abandons a candidate as soon as it determines that the candidate cannot possibly be completed to a valid solution
In the SQL database query language, window functions allow access to data in the records right before and after the current record. A window function defines a frame or window of rows with a given length around the current row, and performs a calculation across the set of data in the window
React is a JavaScript library for building user interfaces. It is maintained by Facebook and a community of individual developers and companies. React can be used as a base in the development of single-page or mobile applications
The fundamental structures of a software system and the discipline of creating such structures and systems. Each structure comprises software elements, relations among them, and properties of both elements and relations.[1] The architecture of a software system is a metaphor, analogous to the architecture of a building.[2] It functions as a blueprint for the system and the developing project, laying out the tasks necessary to be executed by the design teams.
Spark is a library for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. Internally, Spark SQL uses this extra information to perform extra optimizations. There are several ways to interact with Spark SQL including SQL and the Dataset API. When computing a result the same execution engine is used, independent of which API/language you are using to express the computation. This unification means that developers can easily switch back and forth between different APIs based on which provides the most natural way to express a given transformation.
Statistics is a form of mathematical analysis that uses quantified models, representations and synopses for a given set of experimental data or real-life studies. Statistics studies methodologies to gather, review, analyze and draw conclusions from data
A recent book was published this year by Google about site reliability and security engineering, I would like to provide you a brief overview of it and incorporate my own analysis and thoughts about this subject while saving you some time from reading, at least part of it.
On-call, sometimes referred to as on-call shifts are processes used in business where employee work schedules are intentionally unpredictable. Employees who work on-call are expected to be available at any time during their shift, usually with short notice, to carry out their working duties. Depending on the nature of the profession, on-call duties may vary from what is expected during normal working hours to emergency cover only.
While there are a few types of sorting algorithms we have a few shared metrics to evaluate them all. So when you refer me to a specific algorithm we could ask, is this sorting algorithm stable? Is it sorting in place? What is the average worst and best case running time of this sorting algorithm? Which type of data is this sorting algorithm best suited for, how much memory does this sorting algorithm require and does it need actually extra memory?
“The Little Manual of API Design” is a very nice paper written by Jasmin Blanchette has released a paper while working in trolltech, which is a Nokia company. I found it to be very clear and concise, and reassuring what we think of API design. It’s a difficult task that includes both artistic, social, programming and scientific skills. We are going to summarize this paper for you.
In graph theory, a recursive tree is a non-planar labeled rooted tree. A size-n recursive tree is labeled by distinct integers 1, 2, ..., n, where the labels are strictly increasing starting at the root labeled 1. Recursive trees are non-planar, which means that the children of a particular node are not ordered
Brain-machine interfaces (BMIs) hold promise for the restoration of sensory and motor function and the treatment of neurological disorders, but clinical BMIs have not yet been widely adopted, in part because modest channel counts have limited their potential. In this white paper, we describe Neuralink’s first steps toward a scalable high-bandwidth BMI system
Character encoding is used to represent a repertoire of characters by some kind of encoding system. Depending on the abstraction level and context, corresponding code points and the resulting code space may be regarded as bit patterns, octets, natural numbers, electrical pulses
Raft is a consensus algorithm designed as an alternative to Paxos. It was meant to be more understandable than Paxos by means of separation of logic, but it is also formally proven safe and offers some additional features
How could you have a major boost on your programming abilities by having a better understanding of the product. And if you do how much more effort and time would it require from you? what do you need to actually do?
A regular expression, regex or regexp is a sequence of characters that define a search pattern. Usually such patterns are used by string searching algorithms for "find" or "find and replace" operations on strings, or for input validation. It is a technique developed in theoretical computer science and formal language theory
Deciding upon microservices and monolith is not a light decision, it has a high impact on the future of your service, but what about the multimoduled monolith?
Unix File Descriptors and Programmer Abstractions. In Unix and related computer operating systems, a file descriptor is an abstract indicator used to access a file or other input/output resource, such as a pipe or network socket. File descriptors form part of the POSIX application programming interface
In computer science, radix sort is a non-comparative sorting algorithm. It avoids comparison by creating and distributing elements into buckets according to their radix. For elements with more than one significant digit, this bucketing process is repeated for each digit, while preserving the ordering of the prior step, until all digits have been considered
These are the concepts to know when you approach to study the command line tool Awk, it's very powerful but when you try to study it, it many times just does not make sense and appear to be complex, but this should not be the case, this is only because we come from procedural languages, once you grasp the main concepts behind it, it can be much more easier to study it.
Now that you have a micro service, you need to rethink how to handle errors, how do you? Would you do a retry? What if the retry fails, what if the service is too slow and so many clients are now doing retries, the load would increase, how do you handle it then?
We live in a micro services environment and even if not you have for sure some servers some databases and you want to be able to troubleshoot quickly issues, do you really want to scan these long logs and try to find the best way for you to find root cause at night? or do you want to be ready as a professional knowing quickly you already have all the tools for you to troubleshoot.
In this episode I give you one of my biggest tips for implementing and designing distributed systems, should you or not go about retries? what are their disadvantages? what about caches how do they hurt you? what should you avoid when calling databases and other services?
Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a degree of belief in an event. The degree of belief may be based on prior knowledge about the event, such as the results of previous experiments, or on personal beliefs about the event. This differs from a number of other interpretations of probability, such as the frequentist interpretation that views probability as the limit of the relative frequency of an event after many trials.
Queues in System Design Architecture. Queues are about everywhere they are not only a data structure that you study out of the blue in computer science, but they also help distribute your system. But once you distribute your system with queues you face new issues, namely the queue being out of capacity, what do you do in these cases?
002 Scalability Part 2 - Scalibility and startups how to deal with code with databases and scalability and still move fast
Is scalability a mere technological problem? How do you study it? What are the methods to scale a business or tech solution?